[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:40 dist-test-slave-g31s krb5kdc[8251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:40 dist-test-slave-g31s krb5kdc[8251](info): set up 2 sockets
Sep 28 12:30:40 dist-test-slave-g31s krb5kdc[8251](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 12:30:40.998214 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:30:41.128770 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:42323
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:42323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:41.214792 8264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:41.215020 8264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:41.215044 8264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:41.216290 8264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:41.216343 8264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:41.216357 8264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:41.216369 8264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:41.216380 8264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:41.218012 8264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:42323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:42323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:41.218264 8264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:41.218501 8264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:30:41.224027 8264 server_base.cc:1034] running on GCE node
W20260928 12:30:41.224155 8269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:41.224190 8272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:41.224186 8270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:41.224594 8264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:41.224906 8264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:41.226053 8264 hybrid_clock.cc:648] HybridClock initialized: now 1790598641226035 us; error 35 us; skew 500 ppm
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:41.228633 8264 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:30:41.229260 8264 webserver.cc:460] Webserver started at http://127.8.13.62:36509/ using document root <none> and password file <none>
I20260928 12:30:41.229516 8264 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:41.229568 8264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:41.229692 8264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:41.230892 8264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "590b6b23b2db4140a01cd78659d15c8a"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "cc452e1bf2e2eac6403f63084875f3a9"
server_key_iv: "63e6a5a7ac00b791c41c748cdfa633d0"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.231253 8264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "590b6b23b2db4140a01cd78659d15c8a"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "cc452e1bf2e2eac6403f63084875f3a9"
server_key_iv: "63e6a5a7ac00b791c41c748cdfa633d0"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.232775 8264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:30:41.234063 8279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:41.234305 8264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:30:41.234411 8264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "590b6b23b2db4140a01cd78659d15c8a"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "cc452e1bf2e2eac6403f63084875f3a9"
server_key_iv: "63e6a5a7ac00b791c41c748cdfa633d0"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.234490 8264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:41.259485 8264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:41.261955 8264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:41.262107 8264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:41.265944 8264 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:42323
I20260928 12:30:41.265997 8331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:42323 every 8 connection(s)
I20260928 12:30:41.266343 8264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:41.267249 8332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:41.269618 8332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a: Bootstrap starting.
I20260928 12:30:41.270293 8332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:41.270555 8332 log.cc:824] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:41.271389 8332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a: No bootstrap required, opened a new log
I20260928 12:30:41.273227 8332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590b6b23b2db4140a01cd78659d15c8a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42323 } }
I20260928 12:30:41.273399 8332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:41.273423 8332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590b6b23b2db4140a01cd78659d15c8a, State: Initialized, Role: FOLLOWER
I20260928 12:30:41.273618 8332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590b6b23b2db4140a01cd78659d15c8a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42323 } }
I20260928 12:30:41.273686 8332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:41.273707 8332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:41.273741 8332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:41.274214 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 8264
I20260928 12:30:41.274328 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:30:41.274587 8332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590b6b23b2db4140a01cd78659d15c8a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42323 } }
I20260928 12:30:41.274722 8332 leader_election.cc:302] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 590b6b23b2db4140a01cd78659d15c8a; no voters:
I20260928 12:30:41.274919 8332 leader_election.cc:288] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:41.274960 8335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:41.275137 8244 external_mini_cluster.cc:1428] Setting key e66f0431d8c8c0ec6a154922625fd983
I20260928 12:30:41.275164 8335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [term 1 LEADER]: Becoming Leader. State: Replica: 590b6b23b2db4140a01cd78659d15c8a, State: Running, Role: LEADER
I20260928 12:30:41.275197 8332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:41.275255 8335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590b6b23b2db4140a01cd78659d15c8a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42323 } }
I20260928 12:30:41.275766 8337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 590b6b23b2db4140a01cd78659d15c8a. Latest consensus state: current_term: 1 leader_uuid: "590b6b23b2db4140a01cd78659d15c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590b6b23b2db4140a01cd78659d15c8a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42323 } } }
I20260928 12:30:41.275847 8337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:41.275990 8336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "590b6b23b2db4140a01cd78659d15c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590b6b23b2db4140a01cd78659d15c8a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42323 } } }
I20260928 12:30:41.276042 8336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:41.276383 8344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:41.276976 8344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:41.278852 8344 catalog_manager.cc:1347] Generated new cluster ID: c4e2e4ab8f014f31a76a0df986be56c9
I20260928 12:30:41.278936 8344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:41.286435 8344 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:41.287195 8344 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:41.290797 8344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 590b6b23b2db4140a01cd78659d15c8a: Generated new TSK 0
I20260928 12:30:41.291050 8344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:41.296725 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:41.276679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.8.13.62:42323)
0928 12:30:41.276842 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:41.276852 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:41.276887 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:30:41.279359 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:41.279368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:41.279428 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:41.279590 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:41.281765 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.283455 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.285296 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.286033 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.286177 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:41.291490 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:41.291497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:41.291497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:41.291514 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:41.293676 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.294544 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.294550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.294551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.294777 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.295321 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.295323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.295323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.295744 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:41.295970 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:41.296250 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 12:30:41.296471 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:30:41.322067 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42323
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:41.412237 8358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:41.412467 8358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:41.412493 8358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:41.413771 8358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:41.413856 8358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:41.413887 8358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:30:41.415478 8358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:41.415720 8358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:41.415947 8358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:41.420981 8366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:41.421025 8358 server_base.cc:1034] running on GCE node
W20260928 12:30:41.420998 8364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:41.421006 8363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:41.421438 8358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:41.421662 8358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:41.422801 8358 hybrid_clock.cc:648] HybridClock initialized: now 1790598641422784 us; error 31 us; skew 500 ppm
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:41.425362 8358 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:30:41.425848 8358 webserver.cc:460] Webserver started at http://127.8.13.1:34375/ using document root <none> and password file <none>
I20260928 12:30:41.426072 8358 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:41.426117 8358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:41.426257 8358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:41.427389 8358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "131ae61453f340608cbdeced338bfd0b"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "1d823fe72f762333eb602c2ed71553c7"
server_key_iv: "dd775c09d34c54c150d89190ab788ca4"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.427755 8358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "131ae61453f340608cbdeced338bfd0b"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "1d823fe72f762333eb602c2ed71553c7"
server_key_iv: "dd775c09d34c54c150d89190ab788ca4"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.429073 8358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:30:41.429876 8373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:41.430034 8358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:30:41.430132 8358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "131ae61453f340608cbdeced338bfd0b"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "1d823fe72f762333eb602c2ed71553c7"
server_key_iv: "dd775c09d34c54c150d89190ab788ca4"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.430189 8358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:41.457916 8358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:41.459908 8358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:41.460042 8358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:41.460244 8358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:41.460569 8358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:41.460605 8358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:41.460629 8358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:41.460652 8358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:41.466403 8358 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:45879
I20260928 12:30:41.466483 8486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:45879 every 8 connection(s)
I20260928 12:30:41.466797 8358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:30:41.467170 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 8358
I20260928 12:30:41.467250 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:30:41.467404 8244 external_mini_cluster.cc:1428] Setting key 37a815cd055c0919c14a0604fd3f79ed
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
I20260928 12:30:41.477742 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:41.467628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:44069 (local address 127.8.13.62:42323)
0928 12:30:41.467682 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:41.467685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:41.467865 (+ 180us) server_negotiation.cc:407] Connection header received
0928 12:30:41.468382 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:41.468383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:41.468396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:41.468453 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:41.469855 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.470380 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.471603 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.471816 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.471876 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:41.474100 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:41.474101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:41.474102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:41.474111 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:41.475737 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.476408 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.476411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.476412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.476519 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.476993 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.476995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.476995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.477152 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:41.477225 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:41.477536 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 12:30:41.477667 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
I20260928 12:30:41.478037 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:41.467561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:42323 (local address 127.8.13.1:44069)
0928 12:30:41.467748 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:41.467790 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:41.468280 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:41.468511 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:41.468515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:41.469385 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:41.469704 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:41.469713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.470504 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.470505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:41.471490 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:41.471493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.471893 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.471894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:41.472020 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:41.472534 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:41.472543 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:41.473993 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:41.475868 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:41.475886 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:41.475890 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:41.476249 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:41.476695 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:41.476697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:41.476697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:41.476884 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:41.477241 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:41.477249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:41.477403 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 12:30:41.477501 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 12:30:41.477738 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260928 12:30:41.478587 8487 heartbeater.cc:347] Connected to a master server at 127.8.13.62:42323
I20260928 12:30:41.478701 8487 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:41.478909 8487 heartbeater.cc:511] Master 127.8.13.62:42323 requested a full tablet report, sending...
I20260928 12:30:41.479627 8296 ts_manager.cc:194] Registered new tserver with Master: 131ae61453f340608cbdeced338bfd0b (127.8.13.1:45879)
I20260928 12:30:41.480517 8296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:44069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:30:41.496794 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42323
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:41.581351 8492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:41.581604 8492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:41.581632 8492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:41.582983 8492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:41.583043 8492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:41.583076 8492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:30:41.584749 8492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:41.585005 8492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:41.585216 8492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:41.590634 8498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:41.590624 8497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:41.590734 8492 server_base.cc:1034] running on GCE node
W20260928 12:30:41.590626 8500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:41.591109 8492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:41.591378 8492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:41.592548 8492 hybrid_clock.cc:648] HybridClock initialized: now 1790598641592509 us; error 71 us; skew 500 ppm
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:41.595371 8492 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:30:41.595872 8492 webserver.cc:460] Webserver started at http://127.8.13.2:43189/ using document root <none> and password file <none>
I20260928 12:30:41.596114 8492 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:41.596166 8492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:41.596294 8492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:41.597466 8492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "9c1f34f466a84b30ba1edae2ce731f59"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "feda369270df285913b013ba57f17e71"
server_key_iv: "6a6765e3f6956ebeb4ea97ce5b8ef1e9"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.597883 8492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "9c1f34f466a84b30ba1edae2ce731f59"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "feda369270df285913b013ba57f17e71"
server_key_iv: "6a6765e3f6956ebeb4ea97ce5b8ef1e9"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.599273 8492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:30:41.600028 8507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:41.600277 8492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:30:41.600385 8492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "9c1f34f466a84b30ba1edae2ce731f59"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "feda369270df285913b013ba57f17e71"
server_key_iv: "6a6765e3f6956ebeb4ea97ce5b8ef1e9"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.600455 8492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:41.615540 8492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:41.618360 8492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:41.618520 8492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:41.618752 8492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:41.619060 8492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:41.619096 8492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:41.619127 8492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:41.619153 8492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:41.625200 8492 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:36533
I20260928 12:30:41.625267 8620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:36533 every 8 connection(s)
I20260928 12:30:41.625582 8492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:30:41.632330 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 8492
I20260928 12:30:41.632431 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:41.632608 8244 external_mini_cluster.cc:1428] Setting key d4f01cb85af50273399a39907ddb545b
I20260928 12:30:41.636034 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:41.626447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:38261 (local address 127.8.13.62:42323)
0928 12:30:41.626548 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:41.626552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:41.626734 (+ 182us) server_negotiation.cc:407] Connection header received
0928 12:30:41.627413 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:41.627415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:41.627430 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:41.627488 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:41.628848 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.629373 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.630511 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.630715 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.630773 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:41.633067 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:41.633068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:41.633069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:41.633076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:41.634595 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.635048 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.635050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.635050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.635117 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.635360 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.635361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.635361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.635487 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:41.635545 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:41.635834 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 12:30:41.635934 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 12:30:41.636250 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:41.626320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:42323 (local address 127.8.13.2:38261)
0928 12:30:41.626539 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:41.626619 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:41.627291 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:41.627577 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:41.627582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:41.628286 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:41.628642 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:41.628654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.629480 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.629481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:41.630402 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:41.630405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.630791 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.630791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:41.630915 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:41.631463 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:41.631472 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:41.632960 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:41.634692 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:41.634700 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:41.634702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:41.634958 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:41.635179 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:41.635180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:41.635181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:41.635298 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:41.635541 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:41.635548 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:41.635705 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 12:30:41.635806 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 12:30:41.635998 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260928 12:30:41.636718 8621 heartbeater.cc:347] Connected to a master server at 127.8.13.62:42323
I20260928 12:30:41.636838 8621 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:41.637075 8621 heartbeater.cc:511] Master 127.8.13.62:42323 requested a full tablet report, sending...
I20260928 12:30:41.637593 8296 ts_manager.cc:194] Registered new tserver with Master: 9c1f34f466a84b30ba1edae2ce731f59 (127.8.13.2:36533)
I20260928 12:30:41.638823 8296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:38261
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:30:41.662302 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42323
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:41.755959 8626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:41.756300 8626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:41.756347 8626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:41.758841 8626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:41.758930 8626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:41.758982 8626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:30:41.762179 8626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:41.762542 8626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:41.762851 8626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:41.771552 8634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:41.771554 8631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:41.774582 8626 server_base.cc:1034] running on GCE node
W20260928 12:30:41.774828 8632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:41.775098 8626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:41.775480 8626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:41.776597 8626 hybrid_clock.cc:648] HybridClock initialized: now 1790598641776566 us; error 47 us; skew 500 ppm
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:41.780071 8626 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:30:41.780624 8626 webserver.cc:460] Webserver started at http://127.8.13.3:43451/ using document root <none> and password file <none>
I20260928 12:30:41.780849 8626 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:41.780900 8626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:41.781055 8626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:41.782457 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "7383d713163d4be6a6061b0db6c31846"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "5a81b9feac52d0f067aea2a77b374144"
server_key_iv: "5242c2e48648fc0396197794e4f53f03"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.782927 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "7383d713163d4be6a6061b0db6c31846"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "5a81b9feac52d0f067aea2a77b374144"
server_key_iv: "5242c2e48648fc0396197794e4f53f03"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.785506 8626 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 12:30:41.787223 8641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:41.787550 8626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:41.787678 8626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "7383d713163d4be6a6061b0db6c31846"
format_stamp: "Formatted at 2026-09-28 12:30:41 on dist-test-slave-g31s"
server_key: "5a81b9feac52d0f067aea2a77b374144"
server_key_iv: "5242c2e48648fc0396197794e4f53f03"
server_key_version: "encryptionkey@0"
I20260928 12:30:41.787770 8626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:41.803220 8626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:41.805959 8626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:41.806172 8626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:41.806617 8626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:41.807004 8626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:41.807040 8626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:41.807071 8626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:41.807094 8626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:41.822897 8626 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:38161
I20260928 12:30:41.822939 8754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:38161 every 8 connection(s)
I20260928 12:30:41.823287 8626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:30:41.827891 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 8626
I20260928 12:30:41.827970 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:30:41.828097 8244 external_mini_cluster.cc:1428] Setting key 70ab93d48678fada4d84888d511d6b6e
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:41.832551 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:41.823994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:41889 (local address 127.8.13.62:42323)
0928 12:30:41.824046 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:41.824048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:41.824362 (+ 314us) server_negotiation.cc:407] Connection header received
0928 12:30:41.824864 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:41.824865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:41.824878 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:41.824935 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:41.825912 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.826444 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.827394 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.827588 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.827642 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:41.830179 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:41.830180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:41.830181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:41.830188 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:41.831368 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.831752 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.831753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.831753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.831800 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.831977 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.831977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.831977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.832076 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:41.832128 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:41.832406 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 12:30:41.832482 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:30:41.832715 8757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:41.823980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:42323 (local address 127.8.13.3:41889)
0928 12:30:41.824246 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:41.824281 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:41.824773 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:41.825009 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:41.825013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:41.825591 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:41.825824 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:41.825830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.826520 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.826521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:41.827319 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:41.827321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.827652 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.827654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:41.827777 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:41.828290 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:41.828299 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:41.830091 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:41.831451 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:41.831457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:41.831459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:41.831693 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:41.831844 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:41.831844 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:41.831845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:41.831936 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:41.832148 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:41.832154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:41.832280 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 12:30:41.832375 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 12:30:41.832505 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":194,"threads_started":1}
I20260928 12:30:41.833225 8755 heartbeater.cc:347] Connected to a master server at 127.8.13.62:42323
I20260928 12:30:41.833317 8755 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:41.833489 8755 heartbeater.cc:511] Master 127.8.13.62:42323 requested a full tablet report, sending...
I20260928 12:30:41.833881 8296 ts_manager.cc:194] Registered new tserver with Master: 7383d713163d4be6a6061b0db6c31846 (127.8.13.3:38161)
I20260928 12:30:41.834255 8296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:41889
I20260928 12:30:41.840147 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:30:41.864997 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:41.856526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.8.13.1:45879)
0928 12:30:41.856804 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:41.856811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:41.856823 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:30:41.856853 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:41.856855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:41.856870 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:41.856959 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:41.857724 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.858512 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.859262 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.859632 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.859694 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:41.861987 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:41.862000 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:41.862000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:41.862008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:41.863734 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.864081 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.864085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.864085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.864152 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.864319 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.864320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.864320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.864465 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:41.864747 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:41.864798 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 12:30:41.864878 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":77,"threads_started":1}
W20260928 12:30:41.865391 8401 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40234
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:30:41.888700 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:41.880718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.8.13.1:45879)
0928 12:30:41.880774 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:41.880776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:41.880792 (+ 16us) server_negotiation.cc:407] Connection header received
0928 12:30:41.880866 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:41.880867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:41.880880 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:41.880928 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:41.881504 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.882085 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.882934 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:41.883152 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:41.883208 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:41.885526 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:41.885528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:41.885528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:41.885536 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:41.887322 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.887815 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.887818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.887819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.887901 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:41.888170 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:41.888172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:41.888172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:41.888306 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:41.888424 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:41.888517 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 12:30:41.888624 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:30:41.889333 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 8358
I20260928 12:30:41.895437 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 8492
I20260928 12:30:41.901490 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 8626
I20260928 12:30:41.905927 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 8264
2026-09-28T12:30:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1893 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8769](info): set up 2 sockets
Sep 28 12:30:41 dist-test-slave-g31s krb5kdc[8769](info): commencing operation
krb5kdc: starting...
W20260928 12:30:42.873227 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:30:42 dist-test-slave-g31s krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:30:42.996095 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45483
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:45483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:43.079524 8782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:43.079783 8782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:43.079808 8782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:43.081396 8782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:43.081461 8782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:43.081475 8782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:43.081486 8782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:43.081497 8782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:43.083460 8782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:45483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:43.083734 8782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:43.083964 8782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:43.089056 8788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:43.089046 8790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:43.089079 8782 server_base.cc:1034] running on GCE node
W20260928 12:30:43.089046 8787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:43.089507 8782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:43.089900 8782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:43.091142 8782 hybrid_clock.cc:648] HybridClock initialized: now 1790598643091119 us; error 32 us; skew 500 ppm
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598643, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:43.094580 8782 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:30:43.095057 8782 webserver.cc:460] Webserver started at http://127.8.13.62:34119/ using document root <none> and password file <none>
I20260928 12:30:43.095278 8782 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:43.095327 8782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:43.095456 8782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:43.096611 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "bc12dcabe5a54bd4bcb09e7758813568"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "09a140f6ff95ec2947580ddabcefd3ac"
server_key_iv: "e542c6c81740df4bd9416fa6be232051"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.096972 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "bc12dcabe5a54bd4bcb09e7758813568"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "09a140f6ff95ec2947580ddabcefd3ac"
server_key_iv: "e542c6c81740df4bd9416fa6be232051"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.098636 8782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:30:43.099583 8797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:43.099807 8782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:30:43.099910 8782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "bc12dcabe5a54bd4bcb09e7758813568"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "09a140f6ff95ec2947580ddabcefd3ac"
server_key_iv: "e542c6c81740df4bd9416fa6be232051"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.099984 8782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:43.115754 8782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:43.117652 8782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:43.117841 8782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:43.121543 8782 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:45483
I20260928 12:30:43.121589 8849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:45483 every 8 connection(s)
I20260928 12:30:43.121930 8782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:43.122788 8850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:43.125080 8850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568: Bootstrap starting.
I20260928 12:30:43.125710 8850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:43.126044 8850 log.cc:824] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:43.126789 8850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568: No bootstrap required, opened a new log
I20260928 12:30:43.128443 8850 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc12dcabe5a54bd4bcb09e7758813568" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45483 } }
I20260928 12:30:43.128608 8850 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:43.128630 8850 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc12dcabe5a54bd4bcb09e7758813568, State: Initialized, Role: FOLLOWER
I20260928 12:30:43.128769 8850 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc12dcabe5a54bd4bcb09e7758813568" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45483 } }
I20260928 12:30:43.128844 8850 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:43.128868 8850 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:43.128903 8850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:43.129473 8850 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc12dcabe5a54bd4bcb09e7758813568" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45483 } }
I20260928 12:30:43.129572 8850 leader_election.cc:302] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc12dcabe5a54bd4bcb09e7758813568; no voters:
I20260928 12:30:43.129755 8850 leader_election.cc:288] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:43.129837 8853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:43.130036 8850 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:43.130213 8853 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [term 1 LEADER]: Becoming Leader. State: Replica: bc12dcabe5a54bd4bcb09e7758813568, State: Running, Role: LEADER
I20260928 12:30:43.130316 8853 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc12dcabe5a54bd4bcb09e7758813568" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45483 } }
I20260928 12:30:43.130678 8855 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc12dcabe5a54bd4bcb09e7758813568. Latest consensus state: current_term: 1 leader_uuid: "bc12dcabe5a54bd4bcb09e7758813568" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc12dcabe5a54bd4bcb09e7758813568" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45483 } } }
I20260928 12:30:43.130761 8855 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:43.130990 8862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:43.131045 8854 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc12dcabe5a54bd4bcb09e7758813568" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc12dcabe5a54bd4bcb09e7758813568" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45483 } } }
I20260928 12:30:43.131098 8854 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:43.131307 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 8782
I20260928 12:30:43.131387 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:30:43.131520 8244 external_mini_cluster.cc:1428] Setting key 238b6adcd5bfc6036d7227f096c5f986
I20260928 12:30:43.132589 8862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:43.134146 8862 catalog_manager.cc:1347] Generated new cluster ID: b4b1f606097b4d80a53a272f236e7148
I20260928 12:30:43.134200 8862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:43.142141 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.132118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.8.13.62:45483)
0928 12:30:43.132277 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.132285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.132310 (+ 25us) server_negotiation.cc:407] Connection header received
0928 12:30:43.132416 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.132423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.132480 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.132610 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:43.133197 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.134543 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.135286 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.135779 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.135881 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.138218 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:43.138222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:43.138223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:43.138239 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:43.139829 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.140246 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.140250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.140250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.140394 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.140619 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.140620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.140620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.140971 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:43.141599 (+ 628us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.141682 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.141881 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260928 12:30:43.155510 8862 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:43.156181 8862 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:43.165158 8862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc12dcabe5a54bd4bcb09e7758813568: Generated new TSK 0
I20260928 12:30:43.165370 8862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:30:43.196933 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45483
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:43.286017 8876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:43.286249 8876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:43.286278 8876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:43.287638 8876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:43.287705 8876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:43.287737 8876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:30:43.289935 8876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:43.290221 8876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:43.290485 8876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:43.295820 8884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:43.295804 8882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:43.295825 8876 server_base.cc:1034] running on GCE node
W20260928 12:30:43.295807 8881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:43.296241 8876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:43.296525 8876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:43.297721 8876 hybrid_clock.cc:648] HybridClock initialized: now 1790598643297710 us; error 50 us; skew 500 ppm
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598643, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:43.300379 8876 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:30:43.300840 8876 webserver.cc:460] Webserver started at http://127.8.13.1:45903/ using document root <none> and password file <none>
I20260928 12:30:43.301033 8876 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:43.301075 8876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:43.301167 8876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:43.302284 8876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "8b130274e4624183a2873203406dc852"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "6f5516f9daaa5ce71c276f804320f03f"
server_key_iv: "491d32d28fea40bcab388a439b6a5a36"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.302582 8876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "8b130274e4624183a2873203406dc852"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "6f5516f9daaa5ce71c276f804320f03f"
server_key_iv: "491d32d28fea40bcab388a439b6a5a36"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.304123 8876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:43.304870 8891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:43.305024 8876 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:43.305109 8876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "8b130274e4624183a2873203406dc852"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "6f5516f9daaa5ce71c276f804320f03f"
server_key_iv: "491d32d28fea40bcab388a439b6a5a36"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.305173 8876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:43.318863 8876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:43.320647 8876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:43.320760 8876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:43.320953 8876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:43.321219 8876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:43.321251 8876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:43.321272 8876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:43.321282 8876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:43.326869 8876 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:41263
I20260928 12:30:43.326910 9004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:41263 every 8 connection(s)
I20260928 12:30:43.327225 8876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:30:43.331949 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 8876
I20260928 12:30:43.332058 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:30:43.332226 8244 external_mini_cluster.cc:1428] Setting key 457f3cd3f08076cd360d45aa690ada15
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598643, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:43.336802 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.327918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:41801 (local address 127.8.13.62:45483)
0928 12:30:43.327967 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.327969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.328161 (+ 192us) server_negotiation.cc:407] Connection header received
0928 12:30:43.329118 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.329119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.329129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.329188 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:43.329769 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.330328 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.331487 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.331731 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.331782 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.333978 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:43.333979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:43.333980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:43.333987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:43.335378 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.335829 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.335830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.335830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.335888 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.336135 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.336136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.336137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.336254 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:43.336311 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.336601 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.336702 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:30:43.336969 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.327895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45483 (local address 127.8.13.1:41801)
0928 12:30:43.328043 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:43.328084 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:43.329036 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:43.329251 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:43.329255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:43.329507 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:43.329708 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.329711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.330398 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.330399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.331403 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.331405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.331815 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.331816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.331954 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.332494 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:43.332505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:43.333883 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:43.335485 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:43.335495 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:43.335498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:43.335752 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:43.335944 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:43.335945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:43.335946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:43.336075 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:43.336310 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:43.336316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:43.336467 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 12:30:43.336559 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 12:30:43.336735 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260928 12:30:43.337486 9005 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45483
I20260928 12:30:43.337589 9005 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:43.337774 9005 heartbeater.cc:511] Master 127.8.13.62:45483 requested a full tablet report, sending...
I20260928 12:30:43.338428 8814 ts_manager.cc:194] Registered new tserver with Master: 8b130274e4624183a2873203406dc852 (127.8.13.1:41263)
I20260928 12:30:43.339321 8814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:41801
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:30:43.357192 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45483
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:43.441304 9010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:43.441592 9010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:43.441622 9010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:43.443008 9010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:43.443068 9010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:43.443099 9010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:30:43.445164 9010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:43.445405 9010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:43.445606 9010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:43.450635 9018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:43.450635 9016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:43.450661 9015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:43.450730 9010 server_base.cc:1034] running on GCE node
I20260928 12:30:43.451073 9010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:43.451303 9010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:43.452467 9010 hybrid_clock.cc:648] HybridClock initialized: now 1790598643452447 us; error 33 us; skew 500 ppm
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598643, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:43.455302 9010 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:30:43.455786 9010 webserver.cc:460] Webserver started at http://127.8.13.2:39537/ using document root <none> and password file <none>
I20260928 12:30:43.455997 9010 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:43.456059 9010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:43.456192 9010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:43.457279 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "cdfd54ffaeab479aab306c7c3c3f49b1"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "020e48a7404910db0e23384b9f4f9df2"
server_key_iv: "0472ea65f04a069ecf99152c963391c2"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.457636 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "cdfd54ffaeab479aab306c7c3c3f49b1"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "020e48a7404910db0e23384b9f4f9df2"
server_key_iv: "0472ea65f04a069ecf99152c963391c2"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.459028 9010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:43.459892 9025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:43.460076 9010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:43.460180 9010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "cdfd54ffaeab479aab306c7c3c3f49b1"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "020e48a7404910db0e23384b9f4f9df2"
server_key_iv: "0472ea65f04a069ecf99152c963391c2"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.460256 9010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:43.510032 9010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:43.511837 9010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:43.511981 9010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:43.512197 9010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:43.512487 9010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:43.512521 9010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:43.512550 9010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:43.512571 9010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:43.518007 9010 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:38065
I20260928 12:30:43.518054 9138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:38065 every 8 connection(s)
I20260928 12:30:43.518357 9010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:30:43.522678 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 9010
I20260928 12:30:43.522768 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:30:43.522929 8244 external_mini_cluster.cc:1428] Setting key 2824628d6a633af124091261b565b7d8
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598643, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:43.528087 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.519104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:34889 (local address 127.8.13.62:45483)
0928 12:30:43.519148 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.519149 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.519361 (+ 212us) server_negotiation.cc:407] Connection header received
0928 12:30:43.520309 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.520311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.520324 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.520377 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:43.521025 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.521522 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.522536 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.522723 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.522787 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.525174 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:43.525175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:43.525176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:43.525184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:43.526677 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.527116 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.527118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.527118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.527184 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.527417 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.527418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.527418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.527580 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:43.527649 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.527890 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.527982 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:30:43.528149 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.519090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45483 (local address 127.8.13.2:34889)
0928 12:30:43.519243 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:43.519279 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:43.520201 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:43.520449 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:43.520453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:43.520734 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:43.520932 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.520937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.521601 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.521602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.522451 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.522453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.522838 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.522840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.523002 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.523574 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:43.523584 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:43.525082 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:43.526767 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:43.526773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:43.526775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:43.527042 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:43.527247 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:43.527248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:43.527248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:43.527360 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:43.527639 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:43.527642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:43.527767 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 12:30:43.527848 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:30:43.527979 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260928 12:30:43.528661 9139 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45483
I20260928 12:30:43.528767 9139 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:43.528975 9139 heartbeater.cc:511] Master 127.8.13.62:45483 requested a full tablet report, sending...
I20260928 12:30:43.529338 8814 ts_manager.cc:194] Registered new tserver with Master: cdfd54ffaeab479aab306c7c3c3f49b1 (127.8.13.2:38065)
I20260928 12:30:43.529695 8814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:34889
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:30:43.547835 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45483
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:43.629491 9144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:43.629846 9144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:43.629877 9144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:43.631340 9144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:43.631390 9144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:43.631420 9144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:30:43.633086 9144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:43.633338 9144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:43.633581 9144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:43.638653 9150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:43.638644 9149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:43.638644 9152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:43.639017 9144 server_base.cc:1034] running on GCE node
I20260928 12:30:43.639276 9144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:43.639533 9144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:43.640729 9144 hybrid_clock.cc:648] HybridClock initialized: now 1790598643640708 us; error 36 us; skew 500 ppm
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598643, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:43.643651 9144 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:30:43.644127 9144 webserver.cc:460] Webserver started at http://127.8.13.3:35715/ using document root <none> and password file <none>
I20260928 12:30:43.644339 9144 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:43.644384 9144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:43.644513 9144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:43.645705 9144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "694e164ee1a9fbb3b66e4e9d902c0a19"
server_key_iv: "3cb7c9a1e405e22cfd9bfbe6dfdf33c7"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.646107 9144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "694e164ee1a9fbb3b66e4e9d902c0a19"
server_key_iv: "3cb7c9a1e405e22cfd9bfbe6dfdf33c7"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.647509 9144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:43.648351 9159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:43.648514 9144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:43.648630 9144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8"
format_stamp: "Formatted at 2026-09-28 12:30:43 on dist-test-slave-g31s"
server_key: "694e164ee1a9fbb3b66e4e9d902c0a19"
server_key_iv: "3cb7c9a1e405e22cfd9bfbe6dfdf33c7"
server_key_version: "encryptionkey@0"
I20260928 12:30:43.648715 9144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:43.659343 9144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:43.661602 9144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:43.661723 9144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:43.661989 9144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:43.662304 9144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:43.662343 9144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:43.662374 9144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:43.662396 9144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:43.668618 9144 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:37533
I20260928 12:30:43.668658 9272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:37533 every 8 connection(s)
I20260928 12:30:43.668974 9144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:30:43.672735 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 9144
I20260928 12:30:43.672829 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:30:43.672971 8244 external_mini_cluster.cc:1428] Setting key 43643c64cb83d1999c4464b7ba062033
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598643, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:43.678428 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.669683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:49695 (local address 127.8.13.62:45483)
0928 12:30:43.669750 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.669751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.669976 (+ 225us) server_negotiation.cc:407] Connection header received
0928 12:30:43.670957 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.670958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.670972 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.671019 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:43.671679 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.672174 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.673288 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.673507 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.673583 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.675823 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:43.675824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:43.675825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:43.675833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:43.677025 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.677433 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.677434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.677435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.677494 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.677743 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.677744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.677744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.677908 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:43.677967 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.678244 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.678334 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:30:43.678604 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.669655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45483 (local address 127.8.13.3:49695)
0928 12:30:43.669846 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:43.669894 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:43.670841 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:43.671112 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:43.671117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:43.671384 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:43.671589 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.671594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.672261 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.672261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.673210 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.673213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.673596 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.673597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.673732 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.674245 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:43.674253 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:43.675714 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:43.677111 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:43.677120 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:43.677122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:43.677362 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:43.677569 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:43.677570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:43.677570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:43.677687 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:43.677961 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:43.677967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:43.678123 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 12:30:43.678214 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 12:30:43.678379 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260928 12:30:43.679071 9273 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45483
I20260928 12:30:43.679176 9273 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:43.679363 9273 heartbeater.cc:511] Master 127.8.13.62:45483 requested a full tablet report, sending...
I20260928 12:30:43.679734 8814 ts_manager.cc:194] Registered new tserver with Master: b1cd3f7ec6eb4bf0bd5691193022daf8 (127.8.13.3:37533)
I20260928 12:30:43.680086 8814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:49695
I20260928 12:30:43.684883 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:30:43.693089 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.686653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.8.13.62:45483)
0928 12:30:43.686751 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.686753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.686848 (+ 95us) server_negotiation.cc:407] Connection header received
0928 12:30:43.686916 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.686917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.686928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.686991 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:43.687553 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.688049 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.688774 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.688972 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.689026 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.690233 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:43.690234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:43.690235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:43.690243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:43.691771 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.692135 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.692137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.692137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.692195 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.692354 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.692355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.692355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.692515 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:43.692767 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.692892 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.692978 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 12:30:43.695719 8814 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:30:43.697373 8814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:30:43.703907 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.699792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.8.13.3:37533)
0928 12:30:43.699943 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.699947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.700147 (+ 200us) server_negotiation.cc:407] Connection header received
0928 12:30:43.700192 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.700195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.700212 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.700283 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:43.700463 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.701301 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.702813 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.703478 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.703551 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.703619 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.703771 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.703811 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260928 12:30:43.704142 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.699710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.8.13.2:38065)
0928 12:30:43.699880 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.699885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.700159 (+ 274us) server_negotiation.cc:407] Connection header received
0928 12:30:43.700207 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.700210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.700225 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.700296 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:43.700593 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.701656 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.702806 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.703516 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.703589 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.703648 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.703999 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.704052 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260928 12:30:43.703959 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.699607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:38065 (local address 127.0.0.1:52448)
0928 12:30:43.699938 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:43.699980 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:43.700056 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:43.700388 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:43.700389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:43.700391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:43.700452 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.700454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.701784 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.701785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.702712 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.702714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.703687 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.703688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.703756 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.703761 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:30:43.703809 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:30:43.703856 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":63,"thread_start_us":69,"threads_started":1}
I20260928 12:30:43.704371 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.699816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:41263 (local address 127.0.0.1:45906)
0928 12:30:43.700056 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:43.700065 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:43.700106 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:43.700415 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:43.700416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:43.700418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:43.700451 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.700453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.701767 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.701768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.702967 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.702969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.703955 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.703955 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.704012 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.704015 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:30:43.704043 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 12:30:43.704078 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":54,"thread_start_us":64,"threads_started":1}
I20260928 12:30:43.704365 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.700082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.8.13.1:41263)
0928 12:30:43.700252 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.700257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.700265 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:30:43.700300 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.700302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.700316 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.700413 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:43.700496 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.701656 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.703052 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.703813 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.704122 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.704208 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.704236 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.704281 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260928 12:30:43.703958 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.699614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:37533 (local address 127.0.0.1:47136)
0928 12:30:43.699986 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:43.699996 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:43.700069 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:43.700300 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:43.700304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:43.700307 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:43.700387 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.700389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.701414 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.701415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.702711 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.702714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.703559 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.703560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.703659 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.703675 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 12:30:43.703746 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:30:43.703811 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260928 12:30:43.705127 9073 tablet_service.cc:1467] Processing CreateTablet for tablet f50633385df14409be72370a460208ae (DEFAULT_TABLE table=test-table [id=adf504aea04c4c9798eca90236753427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:43.705135 8939 tablet_service.cc:1467] Processing CreateTablet for tablet f50633385df14409be72370a460208ae (DEFAULT_TABLE table=test-table [id=adf504aea04c4c9798eca90236753427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:43.705353 9207 tablet_service.cc:1467] Processing CreateTablet for tablet f50633385df14409be72370a460208ae (DEFAULT_TABLE table=test-table [id=adf504aea04c4c9798eca90236753427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:43.705528 8939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50633385df14409be72370a460208ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:43.705528 9073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50633385df14409be72370a460208ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:43.705693 9207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50633385df14409be72370a460208ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:43.708482 9290 tablet_bootstrap.cc:492] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8: Bootstrap starting.
I20260928 12:30:43.709132 9291 tablet_bootstrap.cc:492] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852: Bootstrap starting.
I20260928 12:30:43.709671 9290 tablet_bootstrap.cc:654] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:43.709820 9291 tablet_bootstrap.cc:654] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:43.709981 9292 tablet_bootstrap.cc:492] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1: Bootstrap starting.
I20260928 12:30:43.710443 9290 log.cc:824] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:43.710450 9291 log.cc:824] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:43.710691 9292 tablet_bootstrap.cc:654] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:43.711175 9292 log.cc:824] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:43.711349 9291 tablet_bootstrap.cc:492] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852: No bootstrap required, opened a new log
I20260928 12:30:43.711421 9290 tablet_bootstrap.cc:492] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8: No bootstrap required, opened a new log
I20260928 12:30:43.711452 9291 ts_tablet_manager.cc:1397] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:30:43.711501 9290 ts_tablet_manager.cc:1397] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 12:30:43.712889 9291 raft_consensus.cc:348] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 38065 } } peers { permanent_uuid: "8b130274e4624183a2873203406dc852" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41263 } } peers { permanent_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37533 } }
I20260928 12:30:43.712889 9290 raft_consensus.cc:348] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 38065 } } peers { permanent_uuid: "8b130274e4624183a2873203406dc852" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41263 } } peers { permanent_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37533 } }
I20260928 12:30:43.713045 9290 raft_consensus.cc:374] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:43.713047 9291 raft_consensus.cc:374] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:43.713068 9290 raft_consensus.cc:729] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1cd3f7ec6eb4bf0bd5691193022daf8, State: Initialized, Role: FOLLOWER
I20260928 12:30:43.713080 9291 raft_consensus.cc:729] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b130274e4624183a2873203406dc852, State: Initialized, Role: FOLLOWER
I20260928 12:30:43.713179 9290 consensus_queue.cc:260] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 38065 } } peers { permanent_uuid: "8b130274e4624183a2873203406dc852" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41263 } } peers { permanent_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37533 } }
I20260928 12:30:43.713227 9291 consensus_queue.cc:260] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 38065 } } peers { permanent_uuid: "8b130274e4624183a2873203406dc852" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41263 } } peers { permanent_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37533 } }
I20260928 12:30:43.713462 9273 heartbeater.cc:503] Master 127.8.13.62:45483 was elected leader, sending a full tablet report...
I20260928 12:30:43.713483 9291 ts_tablet_manager.cc:1428] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:30:43.713510 9005 heartbeater.cc:503] Master 127.8.13.62:45483 was elected leader, sending a full tablet report...
I20260928 12:30:43.713455 9290 ts_tablet_manager.cc:1428] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:30:43.714143 9292 tablet_bootstrap.cc:492] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1: No bootstrap required, opened a new log
I20260928 12:30:43.714210 9292 ts_tablet_manager.cc:1397] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 12:30:43.715837 9292 raft_consensus.cc:348] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 38065 } } peers { permanent_uuid: "8b130274e4624183a2873203406dc852" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41263 } } peers { permanent_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37533 } }
I20260928 12:30:43.716008 9292 raft_consensus.cc:374] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:43.716060 9292 raft_consensus.cc:729] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdfd54ffaeab479aab306c7c3c3f49b1, State: Initialized, Role: FOLLOWER
I20260928 12:30:43.716174 9292 consensus_queue.cc:260] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 38065 } } peers { permanent_uuid: "8b130274e4624183a2873203406dc852" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41263 } } peers { permanent_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37533 } }
I20260928 12:30:43.716434 9292 ts_tablet_manager.cc:1428] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:30:43.716456 9139 heartbeater.cc:503] Master 127.8.13.62:45483 was elected leader, sending a full tablet report...
I20260928 12:30:43.753755 9296 raft_consensus.cc:482] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:30:43.753928 9296 raft_consensus.cc:504] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 38065 } } peers { permanent_uuid: "8b130274e4624183a2873203406dc852" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41263 } } peers { permanent_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37533 } }
I20260928 12:30:43.754261 9296 leader_election.cc:288] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdfd54ffaeab479aab306c7c3c3f49b1 (127.8.13.2:38065), 8b130274e4624183a2873203406dc852 (127.8.13.1:41263)
I20260928 12:30:43.758283 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.754450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:38065 (local address 127.8.13.3:48609)
0928 12:30:43.754494 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:43.754507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:43.754556 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:43.754767 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:43.754768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:43.754770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:43.754841 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.754843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.755789 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.755791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.757102 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.757107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.757963 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.757965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.758070 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.758075 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:30:43.758135 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:30:43.758189 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":33}
I20260928 12:30:43.758282 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.754520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:48609 (local address 127.8.13.2:38065)
0928 12:30:43.754583 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.754585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.754597 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:30:43.754629 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.754630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.754641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.754715 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:43.754908 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.755667 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.757250 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.757878 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.757957 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.757970 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.758155 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.758206 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:30:43.758322 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.754675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:35311 (local address 127.8.13.1:41263)
0928 12:30:43.754727 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.754728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.754802 (+ 74us) server_negotiation.cc:407] Connection header received
0928 12:30:43.754872 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.754873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.754885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.754935 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:43.755060 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.755733 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.757250 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.757879 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.757958 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.757969 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.758206 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.758256 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:30:43.758555 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.754587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:41263 (local address 127.8.13.3:35311)
0928 12:30:43.754749 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:43.754757 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:43.754825 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:43.754950 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:43.754950 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:43.754952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:43.754996 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.754997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.755810 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.755811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.757102 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:43.757107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.758021 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.758022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:43.758115 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.758120 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:30:43.758414 (+ 294us) client_negotiation.cc:240] Negotiation successful
0928 12:30:43.758472 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":25,"thread_start_us":109,"threads_started":1}
I20260928 12:30:43.758693 9093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50633385df14409be72370a460208ae" candidate_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1" is_pre_election: true
I20260928 12:30:43.758852 9093 raft_consensus.cc:2454] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1cd3f7ec6eb4bf0bd5691193022daf8 in term 0.
I20260928 12:30:43.759061 9161 leader_election.cc:302] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1cd3f7ec6eb4bf0bd5691193022daf8, cdfd54ffaeab479aab306c7c3c3f49b1; no voters:
I20260928 12:30:43.759198 9296 raft_consensus.cc:2781] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:30:43.759268 9296 raft_consensus.cc:482] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:30:43.759286 9296 raft_consensus.cc:3037] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:43.759377 8959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50633385df14409be72370a460208ae" candidate_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b130274e4624183a2873203406dc852" is_pre_election: true
I20260928 12:30:43.759487 8959 raft_consensus.cc:2454] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1cd3f7ec6eb4bf0bd5691193022daf8 in term 0.
I20260928 12:30:43.760018 9296 raft_consensus.cc:504] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 38065 } } peers { permanent_uuid: "8b130274e4624183a2873203406dc852" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41263 } } peers { permanent_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37533 } }
I20260928 12:30:43.760145 9296 leader_election.cc:288] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [CANDIDATE]: Term 1 election: Requested vote from peers cdfd54ffaeab479aab306c7c3c3f49b1 (127.8.13.2:38065), 8b130274e4624183a2873203406dc852 (127.8.13.1:41263)
I20260928 12:30:43.760293 9093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50633385df14409be72370a460208ae" candidate_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1"
I20260928 12:30:43.760337 8959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50633385df14409be72370a460208ae" candidate_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b130274e4624183a2873203406dc852"
I20260928 12:30:43.760377 9093 raft_consensus.cc:3037] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:43.760401 8959 raft_consensus.cc:3037] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:43.761087 8959 raft_consensus.cc:2454] T f50633385df14409be72370a460208ae P 8b130274e4624183a2873203406dc852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1cd3f7ec6eb4bf0bd5691193022daf8 in term 1.
I20260928 12:30:43.761099 9093 raft_consensus.cc:2454] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1cd3f7ec6eb4bf0bd5691193022daf8 in term 1.
I20260928 12:30:43.761251 9161 leader_election.cc:302] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b130274e4624183a2873203406dc852, b1cd3f7ec6eb4bf0bd5691193022daf8; no voters:
I20260928 12:30:43.761358 9296 raft_consensus.cc:2781] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:43.761523 9296 raft_consensus.cc:686] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [term 1 LEADER]: Becoming Leader. State: Replica: b1cd3f7ec6eb4bf0bd5691193022daf8, State: Running, Role: LEADER
I20260928 12:30:43.761615 9296 consensus_queue.cc:237] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 38065 } } peers { permanent_uuid: "8b130274e4624183a2873203406dc852" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41263 } } peers { permanent_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37533 } }
I20260928 12:30:43.762498 8814 catalog_manager.cc:5495] T f50633385df14409be72370a460208ae P b1cd3f7ec6eb4bf0bd5691193022daf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1cd3f7ec6eb4bf0bd5691193022daf8 (127.8.13.3). New cstate: current_term: 1 leader_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfd54ffaeab479aab306c7c3c3f49b1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 38065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b130274e4624183a2873203406dc852" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1cd3f7ec6eb4bf0bd5691193022daf8" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37533 } health_report { overall_health: HEALTHY } } }
W20260928 12:30:43.768967 9140 tablet.cc:2369] T f50633385df14409be72370a460208ae P cdfd54ffaeab479aab306c7c3c3f49b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:30:43.782976 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.775082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.8.13.1:41263)
0928 12:30:43.775145 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.775148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.775327 (+ 179us) server_negotiation.cc:407] Connection header received
0928 12:30:43.775356 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.775357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.775368 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.775409 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:43.775954 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.776446 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.777218 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.777425 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.777497 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.780049 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:43.780066 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:43.780066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:43.780075 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:43.781489 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.781904 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.781908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.781908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.781988 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.782250 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.782252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.782252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.782429 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:43.782551 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.782775 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.782863 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598643, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598643, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:30:43.805603 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.798904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.8.13.1:41263)
0928 12:30:43.798972 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.798975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.799041 (+ 66us) server_negotiation.cc:407] Connection header received
0928 12:30:43.799114 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.799115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.799126 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.799173 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:43.799742 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.800224 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.800936 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.801133 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.801189 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.803478 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:43.803479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:43.803479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:43.803486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:43.804607 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.804913 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.804914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.804914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.804962 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.805094 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.805094 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.805094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.805197 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:43.805266 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.805461 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.805524 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:30:43.826216 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:43.820074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.8.13.1:41263)
0928 12:30:43.820145 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:43.820147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:43.820215 (+ 68us) server_negotiation.cc:407] Connection header received
0928 12:30:43.820264 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:43.820265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:43.820275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:43.820322 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:43.820817 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.821292 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.822055 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:43.822234 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:43.822279 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:43.824289 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:43.824291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:43.824291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:43.824298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:43.825344 (+ 1046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.825631 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.825632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.825632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.825676 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:43.825815 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:43.825817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:43.825817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:43.825916 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:43.825988 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:43.826082 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 12:30:43.826146 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 12:30:43.826651 8919 tablet_service.cc:2908] Rejecting scan request for tablet f50633385df14409be72370a460208ae: Uninitialized: clean time has not yet been initialized
I20260928 12:30:43.827481 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 8876
I20260928 12:30:43.832396 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 9010
I20260928 12:30:43.836539 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 9144
I20260928 12:30:43.840619 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 8782
2026-09-28T12:30:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1934 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[9314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[9314](info): set up 2 sockets
Sep 28 12:30:43 dist-test-slave-g31s krb5kdc[9314](info): commencing operation
krb5kdc: starting...
W20260928 12:30:44.805300 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:30:44 dist-test-slave-g31s krb5kdc[9314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:30:44.929203 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:38467
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:38467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:45.014458 9327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:45.014706 9327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:45.014746 9327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:45.016389 9327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:45.016448 9327 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:30:45.016466 9327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:45.016484 9327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:45.016500 9327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:45.016517 9327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:45.018263 9327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:38467
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:45.018548 9327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:45.018782 9327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:45.023468 9335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:45.023554 9333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:45.023651 9327 server_base.cc:1034] running on GCE node
W20260928 12:30:45.023468 9332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:45.023963 9327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:45.024257 9327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:45.025420 9327 hybrid_clock.cc:648] HybridClock initialized: now 1790598645025387 us; error 49 us; skew 500 ppm
Sep 28 12:30:45 dist-test-slave-g31s krb5kdc[9314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598645, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:45.028005 9327 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:30:45.028489 9327 webserver.cc:460] Webserver started at http://127.8.13.62:38635/ using document root <none> and password file <none>
I20260928 12:30:45.028716 9327 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:45.028765 9327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:45.028892 9327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:45.030001 9327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "280e420d5cf64da69ff036d3ceaecd85"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "027145668b2685346106205c43deab8a"
server_key_iv: "38db326e0447f34f4d227c9a506545e0"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.030335 9327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "280e420d5cf64da69ff036d3ceaecd85"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "027145668b2685346106205c43deab8a"
server_key_iv: "38db326e0447f34f4d227c9a506545e0"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.031663 9327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:45.032413 9342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:45.032614 9327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:45.032806 9327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "280e420d5cf64da69ff036d3ceaecd85"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "027145668b2685346106205c43deab8a"
server_key_iv: "38db326e0447f34f4d227c9a506545e0"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.032897 9327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:45.042850 9327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:45.045382 9327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:45.045552 9327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:45.050969 9327 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:38467
I20260928 12:30:45.051016 9404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:38467 every 8 connection(s)
I20260928 12:30:45.051388 9327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:45.052070 9405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:45.054024 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 9327
I20260928 12:30:45.054127 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:30:45.054272 8244 external_mini_cluster.cc:1428] Setting key 285b6f4ca10caf1e4b2c0a7669f481a0
I20260928 12:30:45.054764 9405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85: Bootstrap starting.
I20260928 12:30:45.055467 9405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:45.055753 9405 log.cc:824] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:45.056561 9405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85: No bootstrap required, opened a new log
I20260928 12:30:45.058892 9405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280e420d5cf64da69ff036d3ceaecd85" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38467 } }
I20260928 12:30:45.059074 9405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:45.059110 9405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280e420d5cf64da69ff036d3ceaecd85, State: Initialized, Role: FOLLOWER
I20260928 12:30:45.059283 9405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280e420d5cf64da69ff036d3ceaecd85" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38467 } }
I20260928 12:30:45.059384 9405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:45.059428 9405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:45.059481 9405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:45.060259 9405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280e420d5cf64da69ff036d3ceaecd85" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38467 } }
I20260928 12:30:45.060391 9405 leader_election.cc:302] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 280e420d5cf64da69ff036d3ceaecd85; no voters:
I20260928 12:30:45.060602 9405 leader_election.cc:288] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:45.060660 9410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:45.060825 9410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [term 1 LEADER]: Becoming Leader. State: Replica: 280e420d5cf64da69ff036d3ceaecd85, State: Running, Role: LEADER
I20260928 12:30:45.060904 9405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:45.060920 9410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280e420d5cf64da69ff036d3ceaecd85" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38467 } }
I20260928 12:30:45.061287 9412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 280e420d5cf64da69ff036d3ceaecd85. Latest consensus state: current_term: 1 leader_uuid: "280e420d5cf64da69ff036d3ceaecd85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280e420d5cf64da69ff036d3ceaecd85" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38467 } } }
I20260928 12:30:45.061391 9412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [sys.catalog]: This master's current role is: LEADER
Sep 28 12:30:45 dist-test-slave-g31s krb5kdc[9314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:45.061290 9411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "280e420d5cf64da69ff036d3ceaecd85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280e420d5cf64da69ff036d3ceaecd85" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38467 } } }
I20260928 12:30:45.061542 9411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:45.061950 9416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:45.062562 9416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:45.064610 9416 catalog_manager.cc:1347] Generated new cluster ID: bbdcf97ce0e5400ea6b360ff7a8ea958
I20260928 12:30:45.064669 9416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:45.067152 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.054923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.8.13.62:38467)
0928 12:30:45.055100 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.055109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.055141 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:30:45.055650 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.055661 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.055714 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.055846 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:45.056602 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.057594 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.058676 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.059144 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.059263 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.062100 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:45.062106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:45.062106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:45.062120 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:45.065068 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.065543 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.065546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.065547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.065657 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.065897 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.065898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.065898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.066218 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:45.066367 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.066494 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.066649 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260928 12:30:45.078810 9416 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:45.079402 9416 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:45.084071 9416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85: Generated new TSK 0
I20260928 12:30:45.084285 9416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:30:45.116317 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:38467
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:45.203604 9431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:45.203838 9431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:45.203866 9431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:45.205192 9431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:45.205250 9431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:45.205271 9431 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:30:45.205289 9431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:30:45.206908 9431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.62:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:45.207172 9431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:45.207412 9431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:45.212399 9436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:45.212374 9437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:45.212633 9431 server_base.cc:1034] running on GCE node
W20260928 12:30:45.212380 9439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:45.212913 9431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:45.213153 9431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:45.214318 9431 hybrid_clock.cc:648] HybridClock initialized: now 1790598645214305 us; error 29 us; skew 500 ppm
Sep 28 12:30:45 dist-test-slave-g31s krb5kdc[9314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598645, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:45.221663 9431 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:30:45.222160 9431 webserver.cc:460] Webserver started at http://127.8.13.1:46127/ using document root <none> and password file <none>
I20260928 12:30:45.222368 9431 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:45.222409 9431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:45.222506 9431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:45.223546 9431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "7eb1ef585fb14741a076a54d75d0e2fd"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "3ae35b1e867088d7eea5da56fa2ae668"
server_key_iv: "b85cceb4518d7f7b34dc6ed275a4b0e3"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.223858 9431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "7eb1ef585fb14741a076a54d75d0e2fd"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "3ae35b1e867088d7eea5da56fa2ae668"
server_key_iv: "b85cceb4518d7f7b34dc6ed275a4b0e3"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.225093 9431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:45.225857 9446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:45.226018 9431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:45.226100 9431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "7eb1ef585fb14741a076a54d75d0e2fd"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "3ae35b1e867088d7eea5da56fa2ae668"
server_key_iv: "b85cceb4518d7f7b34dc6ed275a4b0e3"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.226176 9431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:45.244771 9431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:45.247174 9431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:45.247301 9431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:45.247866 9431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:45.247915 9431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:45.247956 9431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:45.247974 9431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:30:45 dist-test-slave-g31s krb5kdc[9314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598645, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:45.254989 9431 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:39063
I20260928 12:30:45.255090 9561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:39063 every 8 connection(s)
I20260928 12:30:45.255378 9431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:30:45.258344 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.248544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:38467 (local address 127.8.13.1:42905)
0928 12:30:45.248720 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:45.248758 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:45.249331 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:45.249660 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:45.249665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:45.250076 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:45.250304 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.250310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.251054 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.251056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.251976 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.251979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.252444 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.252445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.252591 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.253174 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:45.253183 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:45.254989 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:45.256688 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.256695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.256697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.256944 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.257201 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.257202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.257202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.257331 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.257645 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:45.257652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:45.257794 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 12:30:45.257987 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 12:30:45.258164 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260928 12:30:45.258467 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.248542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:42905 (local address 127.8.13.62:38467)
0928 12:30:45.248593 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.248595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.248843 (+ 248us) server_negotiation.cc:407] Connection header received
0928 12:30:45.249436 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.249439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.249452 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.249518 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:45.250408 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.250948 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.252066 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.252362 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.252428 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.255137 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:45.255138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:45.255139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:45.255148 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:45.256584 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.257048 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.257049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.257050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.257113 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.257418 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.257419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.257419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.257557 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:45.257630 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.258174 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.258331 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:30:45.259044 9562 heartbeater.cc:347] Connected to a master server at 127.8.13.62:38467
I20260928 12:30:45.259142 9562 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:45.259322 9562 heartbeater.cc:511] Master 127.8.13.62:38467 requested a full tablet report, sending...
I20260928 12:30:45.259934 9358 ts_manager.cc:194] Registered new tserver with Master: 7eb1ef585fb14741a076a54d75d0e2fd (127.8.13.1:39063)
I20260928 12:30:45.260726 9358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:42905
I20260928 12:30:45.261844 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 9431
I20260928 12:30:45.261946 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:30:45.262153 8244 external_mini_cluster.cc:1428] Setting key 10c97134ac5aa2fdc48ff07cd000cc42
I20260928 12:30:45.267234 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.260372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:49745 (local address 127.8.13.62:38467)
0928 12:30:45.260448 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.260450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.260464 (+ 14us) server_negotiation.cc:407] Connection header received
0928 12:30:45.260535 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.260536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.260547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.260606 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:45.261002 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.261860 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.262840 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.263148 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.263232 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.264573 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:45.264575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:45.264576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:45.264584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:45.265954 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.266391 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.266392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.266393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.266457 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.266652 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.266653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.266653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.266774 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:45.266838 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.267019 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.267126 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:30:45.267262 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.260184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:38467 (local address 127.8.13.1:49745)
0928 12:30:45.260382 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:45.260395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:45.260461 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:45.260610 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:45.260611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:45.260807 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:45.260899 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.260902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.261985 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.261986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.262738 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.262743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.263253 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.263254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.263364 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.263949 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:45.263957 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:45.264437 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:45.266054 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.266056 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.266057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.266289 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.266532 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.266532 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.266533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.266598 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.266857 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:45.266859 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:45.266921 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 12:30:45.266985 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 12:30:45.267128 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:30:45.288831 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:38467
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:45.373627 9571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:45.373957 9571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:45.373992 9571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:45.375428 9571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:45.375489 9571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:45.375514 9571 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:30:45.375533 9571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:30:45.377174 9571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.62:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:45.377429 9571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:45.377660 9571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:45.383025 9579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:45.383031 9576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:45.383068 9571 server_base.cc:1034] running on GCE node
W20260928 12:30:45.382997 9577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:45.383423 9571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:45.383731 9571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:45.384923 9571 hybrid_clock.cc:648] HybridClock initialized: now 1790598645384899 us; error 28 us; skew 500 ppm
Sep 28 12:30:45 dist-test-slave-g31s krb5kdc[9314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598645, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:45.387615 9571 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:30:45.388307 9571 webserver.cc:460] Webserver started at http://127.8.13.2:44735/ using document root <none> and password file <none>
I20260928 12:30:45.388619 9571 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:45.388844 9571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:45.389075 9571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:45.390661 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "34501a2f8a174b49a3d4590489d09380"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "26ab0b8bee40a1e612b352338c1516ef"
server_key_iv: "c717e28b6eb9daeacfaa57293257fef0"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.391332 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "34501a2f8a174b49a3d4590489d09380"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "26ab0b8bee40a1e612b352338c1516ef"
server_key_iv: "c717e28b6eb9daeacfaa57293257fef0"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.392822 9571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:45.393620 9586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:45.393832 9571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:45.393929 9571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "34501a2f8a174b49a3d4590489d09380"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "26ab0b8bee40a1e612b352338c1516ef"
server_key_iv: "c717e28b6eb9daeacfaa57293257fef0"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.394018 9571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:45.404631 9571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:45.406772 9571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:45.406891 9571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:45.407371 9571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:45.407410 9571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:45.407434 9571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:45.407446 9571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:45.413779 9571 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:34065
I20260928 12:30:45.413969 9701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:34065 every 8 connection(s)
I20260928 12:30:45.414209 9571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:30:45 dist-test-slave-g31s krb5kdc[9314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598645, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:45.414354 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 9571
I20260928 12:30:45.414443 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:30:45.414623 8244 external_mini_cluster.cc:1428] Setting key 0c8121a1c46a8bcc38997819a63f3cc5
I20260928 12:30:45.417695 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.407859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:35039 (local address 127.8.13.62:38467)
0928 12:30:45.407912 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.407914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.408133 (+ 219us) server_negotiation.cc:407] Connection header received
0928 12:30:45.408664 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.408666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.408679 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.408745 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:45.409751 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.410578 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.412084 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.412383 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.412460 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.414889 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:45.414890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:45.414891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:45.414899 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:45.416242 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.416660 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.416661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.416661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.416725 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.416949 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.416950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.416951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.417117 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:45.417202 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.417552 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.417627 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:30:45.417958 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.407869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:38467 (local address 127.8.13.2:35039)
0928 12:30:45.408016 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:45.408051 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:45.408539 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:45.409022 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:45.409027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:45.409366 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:45.409651 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.409658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.410679 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.410680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.411994 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.411998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.412500 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.412501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.412648 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.413216 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:45.413225 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:45.414776 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:45.416327 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.416335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.416338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.416593 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.416784 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.416784 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.416784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.416872 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.417226 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:45.417232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:45.417382 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 12:30:45.417502 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 12:30:45.417688 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260928 12:30:45.418509 9702 heartbeater.cc:347] Connected to a master server at 127.8.13.62:38467
I20260928 12:30:45.418599 9702 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:45.418789 9702 heartbeater.cc:511] Master 127.8.13.62:38467 requested a full tablet report, sending...
I20260928 12:30:45.419162 9359 ts_manager.cc:194] Registered new tserver with Master: 34501a2f8a174b49a3d4590489d09380 (127.8.13.2:34065)
I20260928 12:30:45.419600 9359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:35039
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
I20260928 12:30:45.427876 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.422065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:38467 (local address 127.8.13.2:45947)
0928 12:30:45.422193 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:45.422203 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:45.422257 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:45.422384 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:45.422384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:45.422556 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:45.422621 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.422624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.423246 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.423246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.423776 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.423777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.424063 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.424064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.424146 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.424590 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:45.424594 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:45.424951 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:45.426702 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.426704 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.426704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.426874 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.427112 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.427113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.427114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.427187 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.427559 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:45.427560 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:45.427627 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 12:30:45.427697 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:30:45.427795 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260928 12:30:45.427891 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.422140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:45947 (local address 127.8.13.62:38467)
0928 12:30:45.422187 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.422189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.422258 (+ 69us) server_negotiation.cc:407] Connection header received
0928 12:30:45.422299 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.422300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.422311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.422381 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:45.422697 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.423173 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.423833 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.424010 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.424061 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.425025 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:45.425027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:45.425027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:45.425034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:45.426601 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.426948 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.426949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.426949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.427025 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.427256 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.427257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.427257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.427471 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:45.427540 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.427729 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.427808 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:30:45.441152 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:38467
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:45.526114 9711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:45.526360 9711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:45.526389 9711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:45.527607 9711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:45.527647 9711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:45.527666 9711 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:30:45.527683 9711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:30:45.529162 9711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.62:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:45.529385 9711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:45.529589 9711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:45.534369 9719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:45.534427 9717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:45.534475 9711 server_base.cc:1034] running on GCE node
W20260928 12:30:45.534381 9716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:45.534857 9711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:45.535096 9711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:45.536238 9711 hybrid_clock.cc:648] HybridClock initialized: now 1790598645536224 us; error 30 us; skew 500 ppm
Sep 28 12:30:45 dist-test-slave-g31s krb5kdc[9314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598645, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:45.539080 9711 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:30:45.539624 9711 webserver.cc:460] Webserver started at http://127.8.13.3:45367/ using document root <none> and password file <none>
I20260928 12:30:45.539866 9711 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:45.539915 9711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:45.540040 9711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:45.541173 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "16b1d3af6adb44d8823d29e2277c4bf2"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "7b2c356373a914f47c65d3d6721dc422"
server_key_iv: "f8c030e3eb7d1ad193b4c2ca6311cfd0"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.541517 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "16b1d3af6adb44d8823d29e2277c4bf2"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "7b2c356373a914f47c65d3d6721dc422"
server_key_iv: "f8c030e3eb7d1ad193b4c2ca6311cfd0"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.543097 9711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:30:45.544095 9726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:45.544477 9711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:30:45.544603 9711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "16b1d3af6adb44d8823d29e2277c4bf2"
format_stamp: "Formatted at 2026-09-28 12:30:45 on dist-test-slave-g31s"
server_key: "7b2c356373a914f47c65d3d6721dc422"
server_key_iv: "f8c030e3eb7d1ad193b4c2ca6311cfd0"
server_key_version: "encryptionkey@0"
I20260928 12:30:45.544785 9711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:45.561817 9711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:45.564725 9711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:45.564904 9711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:45.565518 9711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:45.565572 9711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:45.565601 9711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:45.565613 9711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:45.572607 9711 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:44317
I20260928 12:30:45.572745 9841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:44317 every 8 connection(s)
I20260928 12:30:45.573000 9711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:30:45 dist-test-slave-g31s krb5kdc[9314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598645, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:45.576548 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.566213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:43613 (local address 127.8.13.62:38467)
0928 12:30:45.566265 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.566267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.566742 (+ 475us) server_negotiation.cc:407] Connection header received
0928 12:30:45.567537 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.567539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.567553 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.567634 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:45.568508 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.569283 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.570986 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.571284 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.571371 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.574099 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:45.574114 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:45.574115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:45.574126 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:45.575254 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.575665 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.575665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.575666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.575714 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.575900 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.575901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.575901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.576017 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:45.576068 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.576405 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.576480 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:30:45.576738 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.566229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:38467 (local address 127.8.13.3:43613)
0928 12:30:45.566572 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:45.566621 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:45.567405 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:45.567729 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:45.567734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:45.568102 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:45.568396 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.568403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.569395 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.569396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.570876 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.570880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.571411 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.571413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.571616 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.572295 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:45.572306 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:45.573979 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:45.575338 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.575345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.575348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.575598 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.575759 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.575759 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.575759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.575852 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.576125 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:45.576135 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:45.576283 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 12:30:45.576380 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 12:30:45.576517 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":229,"threads_started":1}
I20260928 12:30:45.577112 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 9711
I20260928 12:30:45.577198 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:30:45.577183 9842 heartbeater.cc:347] Connected to a master server at 127.8.13.62:38467
I20260928 12:30:45.577261 9842 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:45.577361 8244 external_mini_cluster.cc:1428] Setting key 51061f4959833ede564ff9fc5837ee08
I20260928 12:30:45.577420 9842 heartbeater.cc:511] Master 127.8.13.62:38467 requested a full tablet report, sending...
I20260928 12:30:45.577761 9358 ts_manager.cc:194] Registered new tserver with Master: 16b1d3af6adb44d8823d29e2277c4bf2 (127.8.13.3:44317)
I20260928 12:30:45.578258 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:30:45.578274 9358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:43613
I20260928 12:30:45.585901 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.578453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:57607 (local address 127.8.13.62:38467)
0928 12:30:45.578498 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.578500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.580133 (+ 1633us) server_negotiation.cc:407] Connection header received
0928 12:30:45.580231 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.580232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.580243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.580271 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:45.580683 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.581166 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.582168 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.582366 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.582425 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.583338 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:45.583339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:45.583340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:45.583348 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:45.584772 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.585149 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.585150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.585150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.585200 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.585362 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.585363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.585363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.585463 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:45.585508 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.585719 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.585790 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:30:45.585909 9848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.578349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:38467 (local address 127.8.13.3:57607)
0928 12:30:45.580058 (+ 1709us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:45.580069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:45.580162 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:45.580328 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:45.580329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:45.580552 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:45.580614 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.580617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.581269 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.581270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.582087 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.582091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.582444 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.582445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.582530 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.582942 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:45.582946 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:45.583249 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:45.584851 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.584852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.584853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.585080 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.585248 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:45.585248 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:45.585249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:45.585314 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:45.585510 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:45.585511 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:45.585585 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 12:30:45.585653 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:30:45.585764 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1684,"thread_start_us":76,"threads_started":1}
Sep 28 12:30:45 dist-test-slave-g31s krb5kdc[9314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598645, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:30:45 dist-test-slave-g31s krb5kdc[9314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598645, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:45.602213 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.595103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.8.13.62:38467)
0928 12:30:45.595163 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.595165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.595175 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:30:45.595202 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.595203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.595213 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.595258 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:45.595840 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.596382 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.597142 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.597337 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.597393 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.599648 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:45.599650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:45.599651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:45.599660 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:45.601092 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.601437 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.601438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.601439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.601494 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:45.601657 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:45.601658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:45.601658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:45.601775 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:45.601916 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.602028 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.602118 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:30:45.604053 9358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:30:45.605623 9358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:30:45.612160 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.607899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:44317 (local address 127.0.0.1:46854)
0928 12:30:45.608171 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:45.608183 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:45.608235 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:45.608587 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:45.608590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:45.608605 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:45.608685 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.608688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.610126 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.610127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.611042 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.611044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.611950 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.611951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.612004 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.612007 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:30:45.612052 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:30:45.612082 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260928 12:30:45.612160 9860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.607894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39063 (local address 127.0.0.1:46790)
0928 12:30:45.608070 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:45.608113 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:45.608180 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:45.608586 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:45.608590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:45.608592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:45.608694 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.608696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.610082 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.610083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.611181 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.611183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.611829 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.611830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.611910 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.611924 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:30:45.611989 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:30:45.612045 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":50,"thread_start_us":75,"threads_started":1}
I20260928 12:30:45.612210 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.608238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.8.13.3:44317)
0928 12:30:45.608391 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.608395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.608405 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:30:45.608444 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.608446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.608461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.608542 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:45.608832 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.610036 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.611114 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.611843 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.611947 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.611995 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.612066 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.612104 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260928 12:30:45.612491 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.608006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.8.13.1:39063)
0928 12:30:45.608392 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.608396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.608405 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:30:45.608444 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.608446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.608461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.608542 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:45.608782 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.609976 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.611238 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.611754 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.611820 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.611883 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.612206 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.612416 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":88,"threads_started":1}
I20260928 12:30:45.612730 9862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.608109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:34065 (local address 127.0.0.1:54762)
0928 12:30:45.608633 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:45.608643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:45.608690 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:45.608882 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:45.608882 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:45.608884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:45.608929 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.608930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.609784 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.609785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.611115 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.611117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.612027 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.612028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.612090 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.612095 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:30:45.612143 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:30:45.612186 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":68,"threads_started":1}
I20260928 12:30:45.613057 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.608204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.8.13.2:34065)
0928 12:30:45.608372 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.608377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.608703 (+ 326us) server_negotiation.cc:407] Connection header received
0928 12:30:45.608744 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.608746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.608761 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.608824 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:45.608997 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.609706 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.611186 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.611871 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.611927 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.611995 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.612226 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.612849 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260928 12:30:45.612989 9776 tablet_service.cc:1467] Processing CreateTablet for tablet 410a2fbdca9b4b5f9d7d1deb0744908e (DEFAULT_TABLE table=test-table [id=2c3612ad6da443fb8f3e0ea738e73cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:45.613336 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410a2fbdca9b4b5f9d7d1deb0744908e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:45.613286 9496 tablet_service.cc:1467] Processing CreateTablet for tablet 410a2fbdca9b4b5f9d7d1deb0744908e (DEFAULT_TABLE table=test-table [id=2c3612ad6da443fb8f3e0ea738e73cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:45.613651 9496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410a2fbdca9b4b5f9d7d1deb0744908e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:45.613687 9636 tablet_service.cc:1467] Processing CreateTablet for tablet 410a2fbdca9b4b5f9d7d1deb0744908e (DEFAULT_TABLE table=test-table [id=2c3612ad6da443fb8f3e0ea738e73cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:45.614049 9636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410a2fbdca9b4b5f9d7d1deb0744908e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:45.615927 9865 tablet_bootstrap.cc:492] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: Bootstrap starting.
I20260928 12:30:45.616758 9865 tablet_bootstrap.cc:654] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:45.616986 9865 log.cc:824] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:45.617491 9868 tablet_bootstrap.cc:492] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: Bootstrap starting.
I20260928 12:30:45.618089 9865 tablet_bootstrap.cc:492] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: No bootstrap required, opened a new log
I20260928 12:30:45.618160 9868 tablet_bootstrap.cc:654] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:45.618166 9865 ts_tablet_manager.cc:1397] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:30:45.618468 9868 log.cc:824] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:45.619187 9868 tablet_bootstrap.cc:492] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: No bootstrap required, opened a new log
I20260928 12:30:45.619258 9868 ts_tablet_manager.cc:1397] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:30:45.620113 9865 raft_consensus.cc:348] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } }
I20260928 12:30:45.620291 9865 raft_consensus.cc:374] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:45.620348 9865 raft_consensus.cc:729] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b1d3af6adb44d8823d29e2277c4bf2, State: Initialized, Role: FOLLOWER
I20260928 12:30:45.620514 9865 consensus_queue.cc:260] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } }
I20260928 12:30:45.620790 9865 ts_tablet_manager.cc:1428] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:30:45.620908 9842 heartbeater.cc:503] Master 127.8.13.62:38467 was elected leader, sending a full tablet report...
I20260928 12:30:45.620937 9867 tablet_bootstrap.cc:492] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: Bootstrap starting.
I20260928 12:30:45.621142 9868 raft_consensus.cc:348] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } }
I20260928 12:30:45.621306 9868 raft_consensus.cc:374] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:45.621343 9868 raft_consensus.cc:729] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34501a2f8a174b49a3d4590489d09380, State: Initialized, Role: FOLLOWER
I20260928 12:30:45.621480 9868 consensus_queue.cc:260] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } }
I20260928 12:30:45.621625 9867 tablet_bootstrap.cc:654] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:45.621745 9868 ts_tablet_manager.cc:1428] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:30:45.621920 9702 heartbeater.cc:503] Master 127.8.13.62:38467 was elected leader, sending a full tablet report...
I20260928 12:30:45.622005 9867 log.cc:824] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:45.622792 9867 tablet_bootstrap.cc:492] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: No bootstrap required, opened a new log
I20260928 12:30:45.622864 9867 ts_tablet_manager.cc:1397] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260928 12:30:45.624269 9867 raft_consensus.cc:348] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } }
I20260928 12:30:45.624430 9867 raft_consensus.cc:374] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:45.624467 9867 raft_consensus.cc:729] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb1ef585fb14741a076a54d75d0e2fd, State: Initialized, Role: FOLLOWER
I20260928 12:30:45.624578 9867 consensus_queue.cc:260] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } }
I20260928 12:30:45.624825 9867 ts_tablet_manager.cc:1428] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:30:45.624853 9562 heartbeater.cc:503] Master 127.8.13.62:38467 was elected leader, sending a full tablet report...
W20260928 12:30:45.664834 9703 tablet.cc:2369] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:30:45.756332 9563 tablet.cc:2369] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:30:45.823596 9843 tablet.cc:2369] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:30:45.832775 9873 raft_consensus.cc:482] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:30:45.832907 9873 raft_consensus.cc:504] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } }
I20260928 12:30:45.833225 9873 leader_election.cc:288] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16b1d3af6adb44d8823d29e2277c4bf2 (127.8.13.3:44317), 34501a2f8a174b49a3d4590489d09380 (127.8.13.2:34065)
I20260928 12:30:45.837555 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.833556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:45795 (local address 127.8.13.3:44317)
0928 12:30:45.833619 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.833622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.833732 (+ 110us) server_negotiation.cc:407] Connection header received
0928 12:30:45.833765 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.833766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.833778 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.833860 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:45.834051 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.834897 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.836540 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.837150 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.837217 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.837230 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.837425 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.837468 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:30:45.837553 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.833697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:46643 (local address 127.8.13.2:34065)
0928 12:30:45.833754 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.833756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.833765 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:30:45.833846 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.833847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.833859 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.833908 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:45.834051 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.834897 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.836543 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.837136 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.837217 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.837233 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.837425 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.837468 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:30:45.837620 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.833431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:44317 (local address 127.8.13.1:45795)
0928 12:30:45.833585 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:45.833598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:45.833683 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:45.833864 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:45.833865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:45.833866 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:45.833944 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.833949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.835024 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.835025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.836422 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.836427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.837232 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.837233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.837336 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.837345 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 12:30:45.837414 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 12:30:45.837485 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":63,"thread_start_us":71,"threads_started":1}
I20260928 12:30:45.837630 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.833598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:34065 (local address 127.8.13.1:46643)
0928 12:30:45.833710 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:45.833719 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:45.833760 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:45.833911 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:45.833912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:45.833914 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:45.833945 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.833947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.835024 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.835025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.836422 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:45.836427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.837238 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.837239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:45.837336 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.837344 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 12:30:45.837414 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:30:45.837486 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":89,"thread_start_us":56,"threads_started":1}
I20260928 12:30:45.838007 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410a2fbdca9b4b5f9d7d1deb0744908e" candidate_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" is_pre_election: true
I20260928 12:30:45.838007 9656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410a2fbdca9b4b5f9d7d1deb0744908e" candidate_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34501a2f8a174b49a3d4590489d09380" is_pre_election: true
I20260928 12:30:45.838182 9796 raft_consensus.cc:2454] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb1ef585fb14741a076a54d75d0e2fd in term 0.
I20260928 12:30:45.838182 9656 raft_consensus.cc:2454] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb1ef585fb14741a076a54d75d0e2fd in term 0.
I20260928 12:30:45.838686 9450 leader_election.cc:302] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16b1d3af6adb44d8823d29e2277c4bf2, 7eb1ef585fb14741a076a54d75d0e2fd; no voters:
I20260928 12:30:45.838876 9873 raft_consensus.cc:2781] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:30:45.838949 9873 raft_consensus.cc:482] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:30:45.838976 9873 raft_consensus.cc:3037] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:45.840042 9873 raft_consensus.cc:504] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } }
I20260928 12:30:45.840202 9873 leader_election.cc:288] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 16b1d3af6adb44d8823d29e2277c4bf2 (127.8.13.3:44317), 34501a2f8a174b49a3d4590489d09380 (127.8.13.2:34065)
I20260928 12:30:45.840423 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410a2fbdca9b4b5f9d7d1deb0744908e" candidate_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b1d3af6adb44d8823d29e2277c4bf2"
I20260928 12:30:45.840451 9656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410a2fbdca9b4b5f9d7d1deb0744908e" candidate_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34501a2f8a174b49a3d4590489d09380"
I20260928 12:30:45.840509 9796 raft_consensus.cc:3037] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:45.840554 9656 raft_consensus.cc:3037] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:45.841233 9796 raft_consensus.cc:2454] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb1ef585fb14741a076a54d75d0e2fd in term 1.
I20260928 12:30:45.841301 9656 raft_consensus.cc:2454] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb1ef585fb14741a076a54d75d0e2fd in term 1.
I20260928 12:30:45.841464 9450 leader_election.cc:302] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34501a2f8a174b49a3d4590489d09380, 7eb1ef585fb14741a076a54d75d0e2fd; no voters:
I20260928 12:30:45.841619 9873 raft_consensus.cc:2781] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:45.841768 9873 raft_consensus.cc:686] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 LEADER]: Becoming Leader. State: Replica: 7eb1ef585fb14741a076a54d75d0e2fd, State: Running, Role: LEADER
I20260928 12:30:45.841898 9873 consensus_queue.cc:237] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } }
I20260928 12:30:45.842710 9358 catalog_manager.cc:5495] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb1ef585fb14741a076a54d75d0e2fd (127.8.13.1). New cstate: current_term: 1 leader_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } health_report { overall_health: HEALTHY } } }
I20260928 12:30:45.901176 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:45.898750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.8.13.1:39063)
0928 12:30:45.898803 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:45.898805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:45.898813 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:30:45.898910 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:45.898911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:45.898935 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:45.898988 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:45.899126 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.899608 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.900389 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:45.900583 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:45.900641 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:45.900841 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:45.900945 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:45.901014 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:45.901062 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 12:30:45.901093 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:30:45.903779 9656 raft_consensus.cc:1261] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380 [term 1 FOLLOWER]: Refusing update from remote peer 7eb1ef585fb14741a076a54d75d0e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:45.903784 9796 raft_consensus.cc:1261] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2 [term 1 FOLLOWER]: Refusing update from remote peer 7eb1ef585fb14741a076a54d75d0e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:45.904170 9873 consensus_queue.cc:1035] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:45.904314 9876 consensus_queue.cc:1035] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:45.910869 9358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:45.910986 9358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:45.911728 9358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85: Sending DeleteTablet for 3 replicas of tablet 410a2fbdca9b4b5f9d7d1deb0744908e
I20260928 12:30:45.912097 9496 tablet_service.cc:1514] Processing DeleteTablet for tablet 410a2fbdca9b4b5f9d7d1deb0744908e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:45 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:46790
I20260928 12:30:45.912077 9776 tablet_service.cc:1514] Processing DeleteTablet for tablet 410a2fbdca9b4b5f9d7d1deb0744908e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:45 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:46854
I20260928 12:30:45.912087 9636 tablet_service.cc:1514] Processing DeleteTablet for tablet 410a2fbdca9b4b5f9d7d1deb0744908e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:45 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:54762
I20260928 12:30:45.912322 9890 tablet_replica.cc:330] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: stopping tablet replica
I20260928 12:30:45.912427 9890 raft_consensus.cc:2229] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:45.912546 9890 raft_consensus.cc:2258] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:45.912557 9889 tablet_replica.cc:330] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: stopping tablet replica
I20260928 12:30:45.912611 9358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:30:45.912774 9358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:30:45.912919 9889 raft_consensus.cc:2229] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:45.913022 9890 ts_tablet_manager.cc:1905] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:45.913475 9889 raft_consensus.cc:2258] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:45.913883 9888 tablet_replica.cc:330] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: stopping tablet replica
I20260928 12:30:45.914052 9888 raft_consensus.cc:2229] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:30:45.914208 9636 tablet_service.cc:1467] Processing CreateTablet for tablet cc0d9162def642df97e8d9e8faa59895 (DEFAULT_TABLE table=test-table [id=f781ab895ab34d3e9e6120ef247aaeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:45.914252 9888 raft_consensus.cc:2258] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:45.914422 9636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0d9162def642df97e8d9e8faa59895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:45.915266 9890 ts_tablet_manager.cc:1918] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:30:45.915340 9890 log.cc:1196] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/410a2fbdca9b4b5f9d7d1deb0744908e
I20260928 12:30:45.915468 9868 tablet_bootstrap.cc:492] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380: Bootstrap starting.
I20260928 12:30:45.915601 9890 ts_tablet_manager.cc:1939] T 410a2fbdca9b4b5f9d7d1deb0744908e P 34501a2f8a174b49a3d4590489d09380: Deleting consensus metadata
I20260928 12:30:45.915927 9868 tablet_bootstrap.cc:654] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:45.915997 9346 catalog_manager.cc:4841] TS 34501a2f8a174b49a3d4590489d09380 (127.8.13.2:34065): tablet 410a2fbdca9b4b5f9d7d1deb0744908e (table test-table [id=2c3612ad6da443fb8f3e0ea738e73cf0]) successfully deleted
I20260928 12:30:45.916163 9496 tablet_service.cc:1467] Processing CreateTablet for tablet cc0d9162def642df97e8d9e8faa59895 (DEFAULT_TABLE table=test-table [id=f781ab895ab34d3e9e6120ef247aaeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:45.916266 9496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0d9162def642df97e8d9e8faa59895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:45.916276 9776 tablet_service.cc:1467] Processing CreateTablet for tablet cc0d9162def642df97e8d9e8faa59895 (DEFAULT_TABLE table=test-table [id=f781ab895ab34d3e9e6120ef247aaeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:45.916364 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0d9162def642df97e8d9e8faa59895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:45.917675 9867 tablet_bootstrap.cc:492] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd: Bootstrap starting.
I20260928 12:30:45.919051 9865 tablet_bootstrap.cc:492] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2: Bootstrap starting.
I20260928 12:30:45.919467 9867 tablet_bootstrap.cc:654] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:45.919500 9868 tablet_bootstrap.cc:492] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380: No bootstrap required, opened a new log
I20260928 12:30:45.919544 9868 ts_tablet_manager.cc:1397] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 12:30:45.919605 9865 tablet_bootstrap.cc:654] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:45.919926 9868 raft_consensus.cc:348] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:45.920010 9868 raft_consensus.cc:374] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:45.920030 9868 raft_consensus.cc:729] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34501a2f8a174b49a3d4590489d09380, State: Initialized, Role: FOLLOWER
I20260928 12:30:45.920080 9868 consensus_queue.cc:260] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:45.920168 9868 ts_tablet_manager.cc:1428] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:45.921092 9889 ts_tablet_manager.cc:1905] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:45.921115 9888 ts_tablet_manager.cc:1905] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:45.923729 9889 ts_tablet_manager.cc:1918] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:30:45.923818 9889 log.cc:1196] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/410a2fbdca9b4b5f9d7d1deb0744908e
I20260928 12:30:45.923894 9865 tablet_bootstrap.cc:492] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2: No bootstrap required, opened a new log
I20260928 12:30:45.923930 9888 ts_tablet_manager.cc:1918] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:30:45.923941 9865 ts_tablet_manager.cc:1397] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 12:30:45.924007 9888 log.cc:1196] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/wal/wals/410a2fbdca9b4b5f9d7d1deb0744908e
I20260928 12:30:45.924093 9889 ts_tablet_manager.cc:1939] T 410a2fbdca9b4b5f9d7d1deb0744908e P 16b1d3af6adb44d8823d29e2277c4bf2: Deleting consensus metadata
I20260928 12:30:45.924468 9865 raft_consensus.cc:348] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:45.924559 9865 raft_consensus.cc:374] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:45.924553 9346 catalog_manager.cc:4841] TS 16b1d3af6adb44d8823d29e2277c4bf2 (127.8.13.3:44317): tablet 410a2fbdca9b4b5f9d7d1deb0744908e (table test-table [id=2c3612ad6da443fb8f3e0ea738e73cf0]) successfully deleted
I20260928 12:30:45.924582 9865 raft_consensus.cc:729] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b1d3af6adb44d8823d29e2277c4bf2, State: Initialized, Role: FOLLOWER
I20260928 12:30:45.924621 9865 consensus_queue.cc:260] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:45.924717 9865 ts_tablet_manager.cc:1428] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:45.925010 9888 ts_tablet_manager.cc:1939] T 410a2fbdca9b4b5f9d7d1deb0744908e P 7eb1ef585fb14741a076a54d75d0e2fd: Deleting consensus metadata
I20260928 12:30:45.925057 9867 tablet_bootstrap.cc:492] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd: No bootstrap required, opened a new log
I20260928 12:30:45.925223 9867 ts_tablet_manager.cc:1397] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260928 12:30:45.925599 9343 catalog_manager.cc:4841] TS 7eb1ef585fb14741a076a54d75d0e2fd (127.8.13.1:39063): tablet 410a2fbdca9b4b5f9d7d1deb0744908e (table test-table [id=2c3612ad6da443fb8f3e0ea738e73cf0]) successfully deleted
I20260928 12:30:45.925663 9867 raft_consensus.cc:348] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:45.925752 9867 raft_consensus.cc:374] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:45.925778 9867 raft_consensus.cc:729] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb1ef585fb14741a076a54d75d0e2fd, State: Initialized, Role: FOLLOWER
I20260928 12:30:45.925849 9867 consensus_queue.cc:260] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:45.925941 9867 ts_tablet_manager.cc:1428] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:45.991528 9873 raft_consensus.cc:482] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:30:45.991739 9873 raft_consensus.cc:504] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:45.992092 9873 leader_election.cc:288] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34501a2f8a174b49a3d4590489d09380 (127.8.13.2:34065), 16b1d3af6adb44d8823d29e2277c4bf2 (127.8.13.3:44317)
I20260928 12:30:45.992380 9656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d9162def642df97e8d9e8faa59895" candidate_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34501a2f8a174b49a3d4590489d09380" is_pre_election: true
I20260928 12:30:45.992380 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d9162def642df97e8d9e8faa59895" candidate_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" is_pre_election: true
I20260928 12:30:45.992499 9656 raft_consensus.cc:2454] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb1ef585fb14741a076a54d75d0e2fd in term 0.
I20260928 12:30:45.992499 9796 raft_consensus.cc:2454] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb1ef585fb14741a076a54d75d0e2fd in term 0.
I20260928 12:30:45.992657 9450 leader_election.cc:302] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34501a2f8a174b49a3d4590489d09380, 7eb1ef585fb14741a076a54d75d0e2fd; no voters:
I20260928 12:30:45.992763 9873 raft_consensus.cc:2781] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:30:45.992806 9873 raft_consensus.cc:482] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:30:45.992823 9873 raft_consensus.cc:3037] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:45.993916 9873 raft_consensus.cc:504] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:45.994065 9873 leader_election.cc:288] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 34501a2f8a174b49a3d4590489d09380 (127.8.13.2:34065), 16b1d3af6adb44d8823d29e2277c4bf2 (127.8.13.3:44317)
I20260928 12:30:45.994289 9656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d9162def642df97e8d9e8faa59895" candidate_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34501a2f8a174b49a3d4590489d09380"
I20260928 12:30:45.994379 9656 raft_consensus.cc:3037] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:45.994323 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d9162def642df97e8d9e8faa59895" candidate_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b1d3af6adb44d8823d29e2277c4bf2"
I20260928 12:30:45.994588 9796 raft_consensus.cc:3037] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:45.995276 9656 raft_consensus.cc:2454] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb1ef585fb14741a076a54d75d0e2fd in term 1.
I20260928 12:30:45.995469 9450 leader_election.cc:302] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34501a2f8a174b49a3d4590489d09380, 7eb1ef585fb14741a076a54d75d0e2fd; no voters:
I20260928 12:30:45.995558 9873 raft_consensus.cc:2781] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:45.995606 9873 raft_consensus.cc:686] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 LEADER]: Becoming Leader. State: Replica: 7eb1ef585fb14741a076a54d75d0e2fd, State: Running, Role: LEADER
I20260928 12:30:45.995654 9873 consensus_queue.cc:237] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:45.996058 9796 raft_consensus.cc:2454] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb1ef585fb14741a076a54d75d0e2fd in term 1.
I20260928 12:30:45.996228 9358 catalog_manager.cc:5495] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb1ef585fb14741a076a54d75d0e2fd (127.8.13.1). New cstate: current_term: 1 leader_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } health_report { overall_health: UNKNOWN } } }
Sep 28 12:30:46 dist-test-slave-g31s krb5kdc[9314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598645, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:46.042932 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.034735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:38467 (local address 127.0.0.1:46748)
0928 12:30:46.034995 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:46.035005 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:46.035087 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:46.035207 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:46.035207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:46.035698 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:46.035752 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.035756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.036441 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.036442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.037037 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.037039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.037363 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.037364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.037410 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.038030 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:46.038035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:46.039686 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:46.041630 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:46.041643 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:46.041645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:46.041962 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:46.042271 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:46.042271 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:46.042272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:46.042336 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:46.042592 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:46.042600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:46.042651 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 12:30:46.042708 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:30:46.042827 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":141,"threads_started":1}
I20260928 12:30:46.042986 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.034751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.8.13.62:38467)
0928 12:30:46.034811 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:46.034814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:46.035069 (+ 255us) server_negotiation.cc:407] Connection header received
0928 12:30:46.035146 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:46.035146 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:46.035156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:46.035202 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:46.035833 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.036357 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.037114 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.037306 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.037360 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.039808 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:46.039810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:46.039810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:46.039818 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:46.041463 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:46.042083 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:46.042085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:46.042085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:46.042154 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:46.042389 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:46.042389 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:46.042389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:46.042529 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:46.042579 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:46.042771 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 12:30:46.042882 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":34}
I20260928 12:30:46.045626 9358 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:46748:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:30:46.045882 9358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:30:46.047390 9496 tablet_service.cc:1467] Processing CreateTablet for tablet 97e0117f399742f698070a181e4dfcd7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fac7909730e2493aa517dd03fa1f107c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:30:46.047391 9636 tablet_service.cc:1467] Processing CreateTablet for tablet 97e0117f399742f698070a181e4dfcd7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fac7909730e2493aa517dd03fa1f107c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:30:46.047421 9776 tablet_service.cc:1467] Processing CreateTablet for tablet 97e0117f399742f698070a181e4dfcd7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fac7909730e2493aa517dd03fa1f107c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:30:46.047572 9496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e0117f399742f698070a181e4dfcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:46.047572 9636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e0117f399742f698070a181e4dfcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:46.047582 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e0117f399742f698070a181e4dfcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:46.049225 9867 tablet_bootstrap.cc:492] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd: Bootstrap starting.
I20260928 12:30:46.049389 9868 tablet_bootstrap.cc:492] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: Bootstrap starting.
I20260928 12:30:46.049765 9867 tablet_bootstrap.cc:654] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:46.050258 9868 tablet_bootstrap.cc:654] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:46.050542 9867 tablet_bootstrap.cc:492] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd: No bootstrap required, opened a new log
I20260928 12:30:46.050590 9867 ts_tablet_manager.cc:1397] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:46.050709 9867 raft_consensus.cc:348] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:46.050784 9867 raft_consensus.cc:374] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:46.050819 9867 raft_consensus.cc:729] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb1ef585fb14741a076a54d75d0e2fd, State: Initialized, Role: FOLLOWER
I20260928 12:30:46.050864 9867 consensus_queue.cc:260] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:46.050954 9867 ts_tablet_manager.cc:1428] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:46.051234 9865 tablet_bootstrap.cc:492] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2: Bootstrap starting.
I20260928 12:30:46.052512 9868 tablet_bootstrap.cc:492] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: No bootstrap required, opened a new log
I20260928 12:30:46.052574 9868 ts_tablet_manager.cc:1397] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:30:46.052712 9865 tablet_bootstrap.cc:654] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:46.052994 9868 raft_consensus.cc:348] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:46.053076 9868 raft_consensus.cc:374] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:46.053093 9868 raft_consensus.cc:729] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34501a2f8a174b49a3d4590489d09380, State: Initialized, Role: FOLLOWER
I20260928 12:30:46.053136 9868 consensus_queue.cc:260] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:46.053215 9868 ts_tablet_manager.cc:1428] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:30:46.054133 9871 tablet_replica.cc:439] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } }
I20260928 12:30:46.054318 9871 tablet_replica.cc:442] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:46.054399 9865 tablet_bootstrap.cc:492] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2: No bootstrap required, opened a new log
I20260928 12:30:46.054448 9865 ts_tablet_manager.cc:1397] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:30:46.054524 9876 tablet_replica.cc:439] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } }
I20260928 12:30:46.054637 9876 tablet_replica.cc:442] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:46.054940 9865 raft_consensus.cc:348] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:46.055112 9865 raft_consensus.cc:374] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:46.055153 9865 raft_consensus.cc:729] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b1d3af6adb44d8823d29e2277c4bf2, State: Initialized, Role: FOLLOWER
I20260928 12:30:46.055519 9865 consensus_queue.cc:260] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:46.055647 9865 ts_tablet_manager.cc:1428] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:46.055835 9870 tablet_replica.cc:439] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } }
I20260928 12:30:46.055907 9870 tablet_replica.cc:442] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:46.069995 9871 raft_consensus.cc:482] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:30:46.070101 9871 raft_consensus.cc:504] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:46.070533 9871 leader_election.cc:288] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb1ef585fb14741a076a54d75d0e2fd (127.8.13.1:39063), 16b1d3af6adb44d8823d29e2277c4bf2 (127.8.13.3:44317)
I20260928 12:30:46.074203 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.070796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:44275 (local address 127.8.13.3:44317)
0928 12:30:46.070880 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:46.070881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:46.070991 (+ 110us) server_negotiation.cc:407] Connection header received
0928 12:30:46.071049 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:46.071051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:46.071060 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:46.071129 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:46.071318 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.071884 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.073175 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.073621 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.073678 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.073689 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:46.074066 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 12:30:46.074120 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 12:30:46.074296 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.070732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:44317 (local address 127.8.13.2:44275)
0928 12:30:46.070916 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:46.070927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:46.070993 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:46.071136 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:46.071137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:46.071139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:46.071240 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.071242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.071958 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.071959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.073056 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.073060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.073845 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.073847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.073945 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.073951 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:30:46.074052 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 12:30:46.074109 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260928 12:30:46.074482 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.070820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:59515 (local address 127.8.13.1:39063)
0928 12:30:46.070873 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:46.070875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:46.070995 (+ 120us) server_negotiation.cc:407] Connection header received
0928 12:30:46.071162 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:46.071164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:46.071173 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:46.071201 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:46.071574 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.072113 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.073388 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.073916 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.074006 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.074019 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:46.074352 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 12:30:46.074411 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:30:46.074601 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e0117f399742f698070a181e4dfcd7" candidate_uuid: "34501a2f8a174b49a3d4590489d09380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" is_pre_election: true
I20260928 12:30:46.074703 9796 raft_consensus.cc:2454] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34501a2f8a174b49a3d4590489d09380 in term 0.
I20260928 12:30:46.074810 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.070820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39063 (local address 127.8.13.2:59515)
0928 12:30:46.070920 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:46.070929 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:46.071075 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:46.071264 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:46.071265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:46.071266 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:46.071496 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.071499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.072206 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.072207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.073276 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.073279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.074018 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.074019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.074240 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.074245 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:30:46.074608 (+ 363us) client_negotiation.cc:240] Negotiation successful
0928 12:30:46.074658 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
I20260928 12:30:46.074944 9590 leader_election.cc:302] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16b1d3af6adb44d8823d29e2277c4bf2, 34501a2f8a174b49a3d4590489d09380; no voters:
I20260928 12:30:46.075098 9871 raft_consensus.cc:2781] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:30:46.075141 9871 raft_consensus.cc:482] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:30:46.075158 9871 raft_consensus.cc:3037] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:46.075068 9516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e0117f399742f698070a181e4dfcd7" candidate_uuid: "34501a2f8a174b49a3d4590489d09380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" is_pre_election: true
I20260928 12:30:46.075268 9516 raft_consensus.cc:2454] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34501a2f8a174b49a3d4590489d09380 in term 0.
I20260928 12:30:46.076602 9871 raft_consensus.cc:504] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:46.076737 9871 leader_election.cc:288] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb1ef585fb14741a076a54d75d0e2fd (127.8.13.1:39063), 16b1d3af6adb44d8823d29e2277c4bf2 (127.8.13.3:44317)
I20260928 12:30:46.076948 9516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e0117f399742f698070a181e4dfcd7" candidate_uuid: "34501a2f8a174b49a3d4590489d09380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb1ef585fb14741a076a54d75d0e2fd"
I20260928 12:30:46.077052 9516 raft_consensus.cc:3037] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:46.077109 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e0117f399742f698070a181e4dfcd7" candidate_uuid: "34501a2f8a174b49a3d4590489d09380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b1d3af6adb44d8823d29e2277c4bf2"
I20260928 12:30:46.077183 9796 raft_consensus.cc:3037] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:46.078626 9516 raft_consensus.cc:2454] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34501a2f8a174b49a3d4590489d09380 in term 1.
I20260928 12:30:46.078626 9796 raft_consensus.cc:2454] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34501a2f8a174b49a3d4590489d09380 in term 1.
I20260928 12:30:46.078887 9587 leader_election.cc:302] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34501a2f8a174b49a3d4590489d09380, 7eb1ef585fb14741a076a54d75d0e2fd; no voters:
I20260928 12:30:46.078994 9871 raft_consensus.cc:2781] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:46.079159 9871 raft_consensus.cc:686] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [term 1 LEADER]: Becoming Leader. State: Replica: 34501a2f8a174b49a3d4590489d09380, State: Running, Role: LEADER
I20260928 12:30:46.079248 9871 consensus_queue.cc:237] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } }
I20260928 12:30:46.079612 9906 tablet_replica.cc:439] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: TxnStatusTablet state changed. Reason: New leader 34501a2f8a174b49a3d4590489d09380. Latest consensus state: current_term: 1 leader_uuid: "34501a2f8a174b49a3d4590489d09380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } }
I20260928 12:30:46.079834 9906 tablet_replica.cc:442] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:30:46.080057 9358 catalog_manager.cc:5495] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34501a2f8a174b49a3d4590489d09380 (127.8.13.2). New cstate: current_term: 1 leader_uuid: "34501a2f8a174b49a3d4590489d09380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } health_report { overall_health: UNKNOWN } } }
I20260928 12:30:46.080248 9908 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:30:46.080368 9908 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:30:46.091984 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.088926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.8.13.2:34065)
0928 12:30:46.088990 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:46.088992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:46.089004 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:30:46.089039 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:46.089040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:46.089061 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:46.089139 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:46.089266 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.090120 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.090934 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.091224 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.091300 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.091566 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:46.091716 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:46.091797 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:46.091868 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 12:30:46.091894 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:30:46.092392 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.088735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:34065 (local address 127.0.0.1:54776)
0928 12:30:46.088846 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:46.088860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:46.088929 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:46.089146 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:46.089147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:46.089148 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:30:46.089204 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.089207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.090224 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.090225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.090849 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.090851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.091322 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.091323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.091440 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.091463 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:30:46.091803 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:30:46.091814 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 12:30:46.091862 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:30:46.091914 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 12:30:46.094921 9796 raft_consensus.cc:1261] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 1 FOLLOWER]: Refusing update from remote peer 34501a2f8a174b49a3d4590489d09380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:46.095010 9516 raft_consensus.cc:1261] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 FOLLOWER]: Refusing update from remote peer 34501a2f8a174b49a3d4590489d09380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:46.095304 9906 consensus_queue.cc:1035] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:46.095570 9871 consensus_queue.cc:1035] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:46.096141 9870 tablet_replica.cc:439] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2: TxnStatusTablet state changed. Reason: New leader 34501a2f8a174b49a3d4590489d09380. Latest consensus state: current_term: 1 leader_uuid: "34501a2f8a174b49a3d4590489d09380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } }
I20260928 12:30:46.096335 9870 tablet_replica.cc:442] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:46.096860 9876 tablet_replica.cc:439] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd: TxnStatusTablet state changed. Reason: New leader 34501a2f8a174b49a3d4590489d09380. Latest consensus state: current_term: 1 leader_uuid: "34501a2f8a174b49a3d4590489d09380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } }
I20260928 12:30:46.096999 9876 tablet_replica.cc:442] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:46.097905 9882 mvcc.cc:205] Tried to move back new op lower bound from 7334292054400565248 to 7334292054341206016. Current Snapshot: MvccSnapshot[applied={T|T < 7334292054400565248}]
I20260928 12:30:46.098014 9870 tablet_replica.cc:439] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34501a2f8a174b49a3d4590489d09380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } }
I20260928 12:30:46.098083 9870 tablet_replica.cc:442] T 97e0117f399742f698070a181e4dfcd7 P 16b1d3af6adb44d8823d29e2277c4bf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:46.098152 9876 tablet_replica.cc:439] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34501a2f8a174b49a3d4590489d09380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } }
I20260928 12:30:46.098228 9876 tablet_replica.cc:442] T 97e0117f399742f698070a181e4dfcd7 P 7eb1ef585fb14741a076a54d75d0e2fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:46.097847 9871 tablet_replica.cc:439] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34501a2f8a174b49a3d4590489d09380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } }
I20260928 12:30:46.098667 9871 tablet_replica.cc:442] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:30:46.098930 9880 mvcc.cc:205] Tried to move back new op lower bound from 7334292054400565248 to 7334292054341206016. Current Snapshot: MvccSnapshot[applied={T|T < 7334292054400565248 or (T in {7334292054400565248})}]
I20260928 12:30:46.099423 9871 tablet_replica.cc:439] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34501a2f8a174b49a3d4590489d09380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb1ef585fb14741a076a54d75d0e2fd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39063 } } peers { permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 } } peers { permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 } } }
I20260928 12:30:46.099589 9871 tablet_replica.cc:442] T 97e0117f399742f698070a181e4dfcd7 P 34501a2f8a174b49a3d4590489d09380: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:30:46.106016 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.102749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:34065 (local address 127.8.13.1:42965)
0928 12:30:46.102975 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:46.102986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:46.103055 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:46.103180 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:46.103181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:46.103182 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:30:46.103235 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.103237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.104165 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.104166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.105129 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.105133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.105615 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.105615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.105657 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.105678 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:30:46.105849 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:30:46.105850 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:30:46.105894 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:30:46.105947 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
I20260928 12:30:46.106000 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.102840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:42965 (local address 127.8.13.2:34065)
0928 12:30:46.102886 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:46.102888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:46.103041 (+ 153us) server_negotiation.cc:407] Connection header received
0928 12:30:46.103100 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:46.103101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:46.103112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:46.103177 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:46.103273 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.104057 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.105224 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.105542 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.105604 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.105745 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:46.105782 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:46.105850 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:46.105896 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 12:30:46.105917 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:30:46.108597 9656 raft_consensus.cc:1261] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380 [term 1 FOLLOWER]: Refusing update from remote peer 7eb1ef585fb14741a076a54d75d0e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:46.108600 9796 raft_consensus.cc:1261] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 1 FOLLOWER]: Refusing update from remote peer 7eb1ef585fb14741a076a54d75d0e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:46.108816 9876 consensus_queue.cc:1035] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b1d3af6adb44d8823d29e2277c4bf2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:46.108981 9873 consensus_queue.cc:1035] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "34501a2f8a174b49a3d4590489d09380" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:46.110133 9879 mvcc.cc:205] Tried to move back new op lower bound from 7334292054459142144 to 7334292053998608384. Current Snapshot: MvccSnapshot[applied={T|T < 7334292054459142144}]
I20260928 12:30:46.119199 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.115936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:57407 (local address 127.8.13.1:39063)
0928 12:30:46.115986 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:46.115989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:46.116111 (+ 122us) server_negotiation.cc:407] Connection header received
0928 12:30:46.116176 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:46.116176 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:46.116194 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:46.116250 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:46.116462 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.117016 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.118188 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.118469 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.118556 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.118878 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:46.118918 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:46.118980 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:46.119054 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 12:30:46.119084 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:30:46.119215 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.115861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39063 (local address 127.8.13.2:57407)
0928 12:30:46.116040 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:46.116064 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:46.116128 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:46.116321 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:46.116322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:46.116323 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:30:46.116391 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.116392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.117095 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.117096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.117997 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:46.118006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.118558 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.118560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:46.118664 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.118687 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:30:46.118985 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:30:46.118987 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:30:46.119049 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:30:46.119103 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260928 12:30:46.128710 9358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:46.128837 9358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:46.129477 9358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 280e420d5cf64da69ff036d3ceaecd85: Sending DeleteTablet for 3 replicas of tablet cc0d9162def642df97e8d9e8faa59895
I20260928 12:30:46.129719 9496 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0d9162def642df97e8d9e8faa59895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:46 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:46790
I20260928 12:30:46.129760 9636 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0d9162def642df97e8d9e8faa59895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:46 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:54762
I20260928 12:30:46.129860 9888 tablet_replica.cc:330] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd: stopping tablet replica
I20260928 12:30:46.129912 9888 raft_consensus.cc:2229] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:30:46.129966 9888 raft_consensus.cc:2258] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:46.130039 9890 tablet_replica.cc:330] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380: stopping tablet replica
I20260928 12:30:46.130103 9890 raft_consensus.cc:2229] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:46.130136 9890 raft_consensus.cc:2258] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:46.130501 9890 ts_tablet_manager.cc:1905] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:46.132442 9890 ts_tablet_manager.cc:1918] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:30:46.132501 9890 log.cc:1196] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/cc0d9162def642df97e8d9e8faa59895
I20260928 12:30:46.132632 9888 ts_tablet_manager.cc:1905] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:46.132699 9890 ts_tablet_manager.cc:1939] T cc0d9162def642df97e8d9e8faa59895 P 34501a2f8a174b49a3d4590489d09380: Deleting consensus metadata
I20260928 12:30:46.133167 9346 catalog_manager.cc:4841] TS 34501a2f8a174b49a3d4590489d09380 (127.8.13.2:34065): tablet cc0d9162def642df97e8d9e8faa59895 (table test-table [id=f781ab895ab34d3e9e6120ef247aaeaa]) successfully deleted
I20260928 12:30:46.134229 9888 ts_tablet_manager.cc:1918] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:30:46.134290 9888 log.cc:1196] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-0/wal/wals/cc0d9162def642df97e8d9e8faa59895
I20260928 12:30:46.134485 9888 ts_tablet_manager.cc:1939] T cc0d9162def642df97e8d9e8faa59895 P 7eb1ef585fb14741a076a54d75d0e2fd: Deleting consensus metadata
I20260928 12:30:46.134732 9776 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0d9162def642df97e8d9e8faa59895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:46 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:46854
I20260928 12:30:46.134833 9889 tablet_replica.cc:330] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2: stopping tablet replica
I20260928 12:30:46.134882 9889 raft_consensus.cc:2229] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:46.134917 9889 raft_consensus.cc:2258] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:46.135120 9343 catalog_manager.cc:4841] TS 7eb1ef585fb14741a076a54d75d0e2fd (127.8.13.1:39063): tablet cc0d9162def642df97e8d9e8faa59895 (table test-table [id=f781ab895ab34d3e9e6120ef247aaeaa]) successfully deleted
I20260928 12:30:46.135370 9889 ts_tablet_manager.cc:1905] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:46.136648 9889 ts_tablet_manager.cc:1918] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:30:46.136699 9889 log.cc:1196] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/cc0d9162def642df97e8d9e8faa59895
I20260928 12:30:46.136886 9889 ts_tablet_manager.cc:1939] T cc0d9162def642df97e8d9e8faa59895 P 16b1d3af6adb44d8823d29e2277c4bf2: Deleting consensus metadata
Sep 28 12:30:46 dist-test-slave-g31s krb5kdc[9314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598645, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:30:46.137356 9346 catalog_manager.cc:4841] TS 16b1d3af6adb44d8823d29e2277c4bf2 (127.8.13.3:44317): tablet cc0d9162def642df97e8d9e8faa59895 (table test-table [id=f781ab895ab34d3e9e6120ef247aaeaa]) successfully deleted
I20260928 12:30:46.140429 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.130516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.8.13.1:39063)
0928 12:30:46.130561 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:46.130564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:46.130789 (+ 225us) server_negotiation.cc:407] Connection header received
0928 12:30:46.130838 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:46.130839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:46.130854 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:46.130905 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:46.131567 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.132251 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.133378 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.133653 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.133726 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.137547 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:46.137561 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:46.137562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:46.137571 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:46.139150 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:46.139568 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:46.139573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:46.139573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:46.139652 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:46.139838 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:46.139839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:46.139839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:46.140036 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:46.140198 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:46.140236 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 12:30:46.140328 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 12:30:46.140728 9556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46812
I20260928 12:30:46.147660 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:46.141621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.8.13.62:38467)
0928 12:30:46.141667 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:46.141669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:46.141766 (+ 97us) server_negotiation.cc:407] Connection header received
0928 12:30:46.141834 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:46.141836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:46.141846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:46.141892 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:46.142380 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.142880 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.143698 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:46.143915 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:46.143972 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:46.145146 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:46.145147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:46.145148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:46.145155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:46.146578 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:46.146921 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:46.146922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:46.146923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:46.146978 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:46.147156 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:46.147156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:46.147157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:46.147286 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:46.147372 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:46.147474 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 12:30:46.147556 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 12:30:46.147881 9358 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46762
I20260928 12:30:46.148448 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 9431
I20260928 12:30:46.154518 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 9571
I20260928 12:30:46.160434 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 9711
I20260928 12:30:46.165009 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 9327
2026-09-28T12:30:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2330 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:46 dist-test-slave-g31s krb5kdc[9932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:46 dist-test-slave-g31s krb5kdc[9932](info): set up 2 sockets
Sep 28 12:30:46 dist-test-slave-g31s krb5kdc[9932](info): commencing operation
krb5kdc: starting...
W20260928 12:30:47.138011 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:30:47 dist-test-slave-g31s krb5kdc[9932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:30:47.260479 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45735
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:45735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:47.347827 9945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:47.348176 9945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:47.348219 9945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:47.350618 9945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:47.350720 9945 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:30:47.350744 9945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:47.350765 9945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:47.350782 9945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:47.350797 9945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:47.352813 9945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:45735
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:47.353132 9945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:47.353371 9945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:47.358727 9950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:47.358735 9953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:47.358767 9945 server_base.cc:1034] running on GCE node
W20260928 12:30:47.358727 9951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:47.359315 9945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:47.359629 9945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:47.360822 9945 hybrid_clock.cc:648] HybridClock initialized: now 1790598647360810 us; error 36 us; skew 500 ppm
Sep 28 12:30:47 dist-test-slave-g31s krb5kdc[9932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:47.363878 9945 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:30:47.364704 9945 webserver.cc:460] Webserver started at http://127.8.13.62:33561/ using document root <none> and password file <none>
I20260928 12:30:47.365039 9945 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:47.365103 9945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:47.365278 9945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:47.366829 9945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "8b07aeb9ad324e1193721f3a92b03d83"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "4a17e988f534af92d6768a81bb8e9a1e"
server_key_iv: "cedfeb2a76d1ba46535403925f4e365c"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.367261 9945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "8b07aeb9ad324e1193721f3a92b03d83"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "4a17e988f534af92d6768a81bb8e9a1e"
server_key_iv: "cedfeb2a76d1ba46535403925f4e365c"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.368848 9945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:47.369614 9960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:47.369787 9945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:47.369892 9945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "8b07aeb9ad324e1193721f3a92b03d83"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "4a17e988f534af92d6768a81bb8e9a1e"
server_key_iv: "cedfeb2a76d1ba46535403925f4e365c"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.369973 9945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:47.398393 9945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:47.401407 9945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:47.401577 9945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:47.407400 9945 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:45735
I20260928 12:30:47.407478 10022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:45735 every 8 connection(s)
I20260928 12:30:47.407888 9945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:47.409045 10023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:47.411630 10023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83: Bootstrap starting.
I20260928 12:30:47.412251 10023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:47.412607 10023 log.cc:824] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:47.413367 10023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83: No bootstrap required, opened a new log
I20260928 12:30:47.415596 10023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b07aeb9ad324e1193721f3a92b03d83" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45735 } }
I20260928 12:30:47.415764 10023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:47.415787 10023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b07aeb9ad324e1193721f3a92b03d83, State: Initialized, Role: FOLLOWER
I20260928 12:30:47.415910 10023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b07aeb9ad324e1193721f3a92b03d83" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45735 } }
I20260928 12:30:47.415987 10023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:47.416013 10023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:47.416041 10023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:47.416316 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 9945
I20260928 12:30:47.416393 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:30:47.416528 8244 external_mini_cluster.cc:1428] Setting key 603dc3a2df1e85b8fc5ca0ab91a4b034
I20260928 12:30:47.416671 10023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b07aeb9ad324e1193721f3a92b03d83" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45735 } }
I20260928 12:30:47.416788 10023 leader_election.cc:302] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b07aeb9ad324e1193721f3a92b03d83; no voters:
I20260928 12:30:47.416991 10023 leader_election.cc:288] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:47.417021 10026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:47.417263 10026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [term 1 LEADER]: Becoming Leader. State: Replica: 8b07aeb9ad324e1193721f3a92b03d83, State: Running, Role: LEADER
I20260928 12:30:47.417275 10023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:47.417407 10026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b07aeb9ad324e1193721f3a92b03d83" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45735 } }
I20260928 12:30:47.417894 10028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b07aeb9ad324e1193721f3a92b03d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b07aeb9ad324e1193721f3a92b03d83" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45735 } } }
I20260928 12:30:47.417982 10028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:47.418141 10030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b07aeb9ad324e1193721f3a92b03d83. Latest consensus state: current_term: 1 leader_uuid: "8b07aeb9ad324e1193721f3a92b03d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b07aeb9ad324e1193721f3a92b03d83" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45735 } } }
I20260928 12:30:47.418198 10030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:47.418304 10037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:47.418874 10037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:47.420819 10037 catalog_manager.cc:1347] Generated new cluster ID: 5080667655c34cbd845e669337369162
I20260928 12:30:47.420876 10037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:30:47 dist-test-slave-g31s krb5kdc[9932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:47.428013 10037 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:47.428714 10037 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:47.428967 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.417026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.8.13.62:45735)
0928 12:30:47.417351 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:47.417362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:47.417397 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:30:47.417529 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:47.417537 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:47.417597 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:47.417740 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:47.418875 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.420270 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.421371 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.421999 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.422123 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.424878 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:47.424884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:47.424884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:47.424900 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:47.426988 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.427419 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.427423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.427424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.427540 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.427761 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.427762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.427763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.428067 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:47.428229 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:47.428484 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 12:30:47.428705 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":279,"thread_start_us":97,"threads_started":1}
I20260928 12:30:47.432170 10037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83: Generated new TSK 0
I20260928 12:30:47.432369 10037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:30:47.461385 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45735
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:47.549309 10049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:47.549590 10049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:47.549624 10049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:47.551422 10049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:47.551483 10049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:47.551507 10049 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:30:47.551517 10049 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:30:47.551544 10049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:30:47.553318 10049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.62:45735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:47.553593 10049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:47.553864 10049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:47.559351 10057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:47.559327 10055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:47.559502 10054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:47.559422 10049 server_base.cc:1034] running on GCE node
I20260928 12:30:47.559751 10049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:47.559979 10049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:47.561133 10049 hybrid_clock.cc:648] HybridClock initialized: now 1790598647561117 us; error 36 us; skew 500 ppm
Sep 28 12:30:47 dist-test-slave-g31s krb5kdc[9932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:47.564392 10049 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:30:47.565047 10049 webserver.cc:460] Webserver started at http://127.8.13.1:33547/ using document root <none> and password file <none>
I20260928 12:30:47.565281 10049 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:47.565336 10049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:47.565491 10049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:47.566849 10049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "f56b1f0e179e4cef8520a19aff4b4e30"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "2be0f7969253ec3fc78004940ffbac0e"
server_key_iv: "c7134d4a722b04740feab59867b32b49"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.567276 10049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "f56b1f0e179e4cef8520a19aff4b4e30"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "2be0f7969253ec3fc78004940ffbac0e"
server_key_iv: "c7134d4a722b04740feab59867b32b49"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.568815 10049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:30:47.569757 10064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:47.570098 10049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:47.570222 10049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "f56b1f0e179e4cef8520a19aff4b4e30"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "2be0f7969253ec3fc78004940ffbac0e"
server_key_iv: "c7134d4a722b04740feab59867b32b49"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.570303 10049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:47.586941 10049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:47.589273 10049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:47.589450 10049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:47.590080 10049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:47.590124 10049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:47.590157 10049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:47.590173 10049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:47.597180 10049 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:38793
Sep 28 12:30:47 dist-test-slave-g31s krb5kdc[9932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:47.597448 10179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:38793 every 8 connection(s)
I20260928 12:30:47.597640 10049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:30:47.597832 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 10049
I20260928 12:30:47.597946 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:30:47.598134 8244 external_mini_cluster.cc:1428] Setting key 01caddbcb879c615edaa2ebe25d18624
I20260928 12:30:47.601002 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.590547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:36141 (local address 127.8.13.62:45735)
0928 12:30:47.590593 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:47.590595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:47.590821 (+ 226us) server_negotiation.cc:407] Connection header received
0928 12:30:47.591343 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:47.591345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:47.591358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:47.591410 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:47.592228 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.593001 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.594722 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.595038 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.595121 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.598098 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:47.598100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:47.598100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:47.598121 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:47.599596 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.600099 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.600100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.600101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.600170 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.600378 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.600379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.600379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.600518 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:47.600584 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:47.600825 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 12:30:47.600909 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:30:47.601191 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.590559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45735 (local address 127.8.13.1:36141)
0928 12:30:47.590701 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:47.590737 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:47.591243 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:47.591480 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:47.591484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:47.591835 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:47.592139 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.592145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.593134 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.593135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.594607 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.594612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.595137 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.595138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.595304 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.596005 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:47.596023 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:47.597960 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:47.599684 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.599692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.599695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.600017 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.600231 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.600231 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.600232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.600327 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.600574 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:47.600577 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:47.600703 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 12:30:47.600784 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:30:47.600955 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260928 12:30:47.601677 10180 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45735
I20260928 12:30:47.601759 10180 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:47.601975 10180 heartbeater.cc:511] Master 127.8.13.62:45735 requested a full tablet report, sending...
I20260928 12:30:47.602533 9976 ts_manager.cc:194] Registered new tserver with Master: f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1:38793)
I20260928 12:30:47.603398 9976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:36141
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
I20260928 12:30:47.608901 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.602707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:57227 (local address 127.8.13.62:45735)
0928 12:30:47.602745 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:47.602746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:47.602899 (+ 153us) server_negotiation.cc:407] Connection header received
0928 12:30:47.602997 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:47.602998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:47.603007 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:47.603032 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:47.603424 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.604062 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.604791 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.604992 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.605047 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.606035 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:47.606036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:47.606037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:47.606044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:47.607266 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.607609 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.607610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.607610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.607669 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.607902 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.607903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.607904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.608071 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:47.608140 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:47.608639 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 12:30:47.608783 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:30:47.609285 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.602716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45735 (local address 127.8.13.1:57227)
0928 12:30:47.602839 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:47.602849 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:47.602922 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:47.603088 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:47.603089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:47.603278 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:47.603349 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.603351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.604171 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.604172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.604712 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.604714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.605051 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.605052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.605126 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.605568 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:47.605572 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:47.605967 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:47.607347 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.607348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.607349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.607523 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.607748 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.607749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.607749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.607828 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.608185 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:47.608187 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:47.608357 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 12:30:47.608577 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 12:30:47.609162 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:30:47.624397 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45735
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:47.708273 10189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:47.708496 10189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:47.708523 10189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:47.709681 10189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:47.709719 10189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:47.709738 10189 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:30:47.709749 10189 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:30:47.709764 10189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:30:47.712069 10189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.62:45735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:47.712308 10189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:47.712504 10189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:47.717389 10197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:47.717598 10194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:47.717628 10189 server_base.cc:1034] running on GCE node
W20260928 12:30:47.717432 10195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:47.718137 10189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:47.718415 10189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:47.719568 10189 hybrid_clock.cc:648] HybridClock initialized: now 1790598647719546 us; error 37 us; skew 500 ppm
Sep 28 12:30:47 dist-test-slave-g31s krb5kdc[9932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:47.721993 10189 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:30:47.722422 10189 webserver.cc:460] Webserver started at http://127.8.13.2:33443/ using document root <none> and password file <none>
I20260928 12:30:47.722633 10189 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:47.722680 10189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:47.722805 10189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:47.724156 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "98a465a63a474d0db37dc956aa6788a1"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "1641c6ff168c6646012334ea8be7bf1d"
server_key_iv: "31793f87f1d8d4ac69901ffda109e8ff"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.724574 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "98a465a63a474d0db37dc956aa6788a1"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "1641c6ff168c6646012334ea8be7bf1d"
server_key_iv: "31793f87f1d8d4ac69901ffda109e8ff"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.725900 10189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:30:47.726658 10204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:47.726825 10189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:30:47.726914 10189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "98a465a63a474d0db37dc956aa6788a1"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "1641c6ff168c6646012334ea8be7bf1d"
server_key_iv: "31793f87f1d8d4ac69901ffda109e8ff"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.726977 10189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:47.742702 10189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:47.745486 10189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:47.745646 10189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:47.746188 10189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:47.746233 10189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:47.746265 10189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:47.746282 10189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:47.753060 10189 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:45519
I20260928 12:30:47.753136 10319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:45519 every 8 connection(s)
Sep 28 12:30:47 dist-test-slave-g31s krb5kdc[9932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:47.753473 10189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:30:47.756668 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.746660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:45605 (local address 127.8.13.62:45735)
0928 12:30:47.746737 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:47.746739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:47.746928 (+ 189us) server_negotiation.cc:407] Connection header received
0928 12:30:47.747471 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:47.747472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:47.747485 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:47.747536 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:47.748442 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.749235 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.750550 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.750854 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.750943 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.753838 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:47.753840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:47.753841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:47.753851 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:47.755300 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.755794 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.755794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.755795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.755845 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.756052 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.756053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.756053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.756167 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:47.756213 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:47.756518 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 12:30:47.756595 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:30:47.756876 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.746647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45735 (local address 127.8.13.2:45605)
0928 12:30:47.746807 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:47.746849 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:47.747373 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:47.747614 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:47.747619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:47.747993 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:47.748319 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.748325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.749381 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.749382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.750466 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.750469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.750965 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.750966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.751129 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.752021 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:47.752032 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:47.753674 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:47.755404 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.755412 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.755415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.755708 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.755892 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.755893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.755893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.755997 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.756247 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:47.756253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:47.756398 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 12:30:47.756500 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 12:30:47.756654 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260928 12:30:47.757326 10320 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45735
I20260928 12:30:47.757408 10320 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:47.757589 10320 heartbeater.cc:511] Master 127.8.13.62:45735 requested a full tablet report, sending...
I20260928 12:30:47.758000 9977 ts_manager.cc:194] Registered new tserver with Master: 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519)
I20260928 12:30:47.758498 9977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:45605
I20260928 12:30:47.759337 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 10189
I20260928 12:30:47.759436 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:30:47.759639 8244 external_mini_cluster.cc:1428] Setting key 3c6becd53ca64c6c2b091ec0a1cd9537
I20260928 12:30:47.765112 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.758371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45735 (local address 127.8.13.2:35893)
0928 12:30:47.758512 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:47.758522 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:47.758577 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:47.758846 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:47.758849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:47.759158 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:47.759258 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.759261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.760205 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.760207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.760807 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.760810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.761151 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.761152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.761233 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.761712 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:47.761717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:47.762095 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:47.763905 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.763907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.763908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.764131 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.764367 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.764368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.764368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.764432 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.764797 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:47.764798 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:47.764846 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 12:30:47.764903 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:30:47.765019 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260928 12:30:47.765122 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.758428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:35893 (local address 127.8.13.62:45735)
0928 12:30:47.758468 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:47.758469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:47.758572 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:30:47.758684 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:47.758684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:47.758696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:47.758752 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:47.759409 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.760096 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.760885 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.761085 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.761140 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.762179 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:47.762180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:47.762181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:47.762189 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:47.763784 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.764207 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.764209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.764209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.764274 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.764495 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.764496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.764496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.764722 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:47.764798 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:47.764943 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 12:30:47.765026 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:30:47.789755 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45735
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:47.875347 10329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:47.875573 10329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:47.875612 10329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:47.876928 10329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:47.876984 10329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:47.877017 10329 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:30:47.877033 10329 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:30:47.877058 10329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:30:47.879045 10329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.62:45735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:47.879343 10329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:47.879591 10329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:47.884739 10335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:47.884737 10334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:47.884749 10337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:47.884959 10329 server_base.cc:1034] running on GCE node
I20260928 12:30:47.885213 10329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:47.885461 10329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:47.886631 10329 hybrid_clock.cc:648] HybridClock initialized: now 1790598647886614 us; error 58 us; skew 500 ppm
Sep 28 12:30:47 dist-test-slave-g31s krb5kdc[9932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:47.889699 10329 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:30:47.890203 10329 webserver.cc:460] Webserver started at http://127.8.13.3:41751/ using document root <none> and password file <none>
I20260928 12:30:47.890441 10329 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:47.890487 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:47.890616 10329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:47.891690 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "ae68cbe1e42642e98af1ac534571b116"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "db200c6bd33e6898bfcf91264828d050"
server_key_iv: "9d01c15b1480be670cc5dd8b3a6b9ab6"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.892031 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "ae68cbe1e42642e98af1ac534571b116"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "db200c6bd33e6898bfcf91264828d050"
server_key_iv: "9d01c15b1480be670cc5dd8b3a6b9ab6"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.893396 10329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:47.894196 10344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:47.894362 10329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:47.894459 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "ae68cbe1e42642e98af1ac534571b116"
format_stamp: "Formatted at 2026-09-28 12:30:47 on dist-test-slave-g31s"
server_key: "db200c6bd33e6898bfcf91264828d050"
server_key_iv: "9d01c15b1480be670cc5dd8b3a6b9ab6"
server_key_version: "encryptionkey@0"
I20260928 12:30:47.894542 10329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:47.917245 10329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:47.919538 10329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:47.919652 10329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:47.920156 10329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:47.920197 10329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:47.920221 10329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:47.920243 10329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:30:47 dist-test-slave-g31s krb5kdc[9932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:47.926906 10329 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:32847
I20260928 12:30:47.926972 10459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:32847 every 8 connection(s)
I20260928 12:30:47.927305 10329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:30:47.929198 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.920576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:52935 (local address 127.8.13.62:45735)
0928 12:30:47.920642 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:47.920644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:47.920821 (+ 177us) server_negotiation.cc:407] Connection header received
0928 12:30:47.921305 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:47.921306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:47.921320 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:47.921395 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:47.922243 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.922994 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.924039 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.924327 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.924419 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.926739 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:47.926741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:47.926741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:47.926749 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:47.928074 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.928445 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.928446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.928446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.928494 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.928677 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.928678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.928678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.928793 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:47.928855 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:47.929073 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 12:30:47.929136 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:30:47.929358 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.920555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45735 (local address 127.8.13.3:52935)
0928 12:30:47.920699 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:47.920738 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:47.921213 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:47.921477 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:47.921487 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:47.921843 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:47.922131 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.922138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.923099 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.923100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.923966 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.923969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.924439 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.924440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.924574 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.925151 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:47.925159 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:47.926636 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:47.928143 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.928146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.928148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.928384 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.928542 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.928543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.928543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.928635 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.928847 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:47.928850 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:47.928958 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 12:30:47.929041 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:30:47.929164 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260928 12:30:47.929816 10460 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45735
I20260928 12:30:47.929895 10460 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:47.930078 10460 heartbeater.cc:511] Master 127.8.13.62:45735 requested a full tablet report, sending...
I20260928 12:30:47.930437 9976 ts_manager.cc:194] Registered new tserver with Master: ae68cbe1e42642e98af1ac534571b116 (127.8.13.3:32847)
I20260928 12:30:47.930912 9976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:52935
I20260928 12:30:47.935014 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 10329
I20260928 12:30:47.935098 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:30:47.935267 8244 external_mini_cluster.cc:1428] Setting key f10a2641f91442b295e5bb0c6202fa7a
I20260928 12:30:47.936157 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:30:47.936695 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.930856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:56347 (local address 127.8.13.62:45735)
0928 12:30:47.930893 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:47.930895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:47.930955 (+ 60us) server_negotiation.cc:407] Connection header received
0928 12:30:47.931035 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:47.931036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:47.931045 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:47.931094 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:47.931407 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.932121 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.933026 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.933206 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.933260 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.934186 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:47.934188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:47.934188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:47.934194 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:47.935602 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.935971 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.935972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.935972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.936037 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.936223 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.936224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.936224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.936351 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:47.936401 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:47.936552 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 12:30:47.936631 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:30:47.936757 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.930784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45735 (local address 127.8.13.3:56347)
0928 12:30:47.930899 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:47.930910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:47.930970 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:47.931098 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:47.931099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:47.931291 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:47.931342 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.931343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.932325 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.932326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.932941 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.932943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.933268 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.933269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.933325 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.933729 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:47.933733 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:47.934084 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:47.935676 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.935678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.935678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.935907 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.936098 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:47.936099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:47.936099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:47.936163 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:47.936406 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:47.936406 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:47.936456 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 12:30:47.936533 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 12:30:47.936643 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
Sep 28 12:30:47 dist-test-slave-g31s krb5kdc[9932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:30:47 dist-test-slave-g31s krb5kdc[9932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:47.959821 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.952849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.8.13.62:45735)
0928 12:30:47.952903 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:47.952905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:47.952948 (+ 43us) server_negotiation.cc:407] Connection header received
0928 12:30:47.952974 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:47.952975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:47.952983 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:47.953024 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:47.953640 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.954209 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.955031 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.955229 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.955294 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.957656 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:47.957658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:47.957658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:47.957666 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:47.958931 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.959261 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.959261 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.959262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.959305 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:47.959432 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:47.959432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:47.959432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:47.959539 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:47.959609 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:47.959687 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:30:47.959748 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:30:47.961544 9976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:30:47.963333 9976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:30:47.969714 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.965726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:32847 (local address 127.0.0.1:53454)
0928 12:30:47.965852 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:47.965892 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:47.965955 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:47.966240 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:47.966243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:47.966247 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:47.966319 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.966321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.967576 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.967577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.968571 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.968574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.969369 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.969370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.969464 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.969479 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 12:30:47.969558 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 12:30:47.969620 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":29,"thread_start_us":45,"threads_started":1}
I20260928 12:30:47.969727 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.965873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.8.13.3:32847)
0928 12:30:47.966066 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:47.966072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:47.966084 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:30:47.966125 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:47.966127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:47.966143 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:47.966230 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:47.966388 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.967487 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.968671 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.969285 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.969357 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.969417 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:47.969578 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 12:30:47.969624 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260928 12:30:47.970019 10479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.966091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:45519 (local address 127.0.0.1:58528)
0928 12:30:47.966325 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:47.966332 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:47.966363 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:47.966602 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:47.966603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:47.966605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:47.966653 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.966655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.967831 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.967832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.968952 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.968954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.969779 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.969780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.969863 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.969868 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:30:47.969913 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:30:47.969958 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":27,"thread_start_us":67,"threads_started":1}
I20260928 12:30:47.970110 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.965937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:38793 (local address 127.0.0.1:58284)
0928 12:30:47.966206 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:47.966215 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:47.966260 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:47.966803 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:47.966804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:47.966806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:47.966847 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.966849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.967725 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.967726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.968954 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:47.968955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.969866 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.969867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:47.969941 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.969947 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:30:47.969998 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:30:47.970042 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":21,"thread_start_us":76,"threads_started":1}
I20260928 12:30:47.970431 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.966201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.8.13.2:45519)
0928 12:30:47.966361 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:47.966365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:47.966374 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:30:47.966423 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:47.966425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:47.966440 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:47.966504 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:47.966732 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.967752 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.969045 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.969641 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.969698 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.969747 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:47.970261 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 12:30:47.970307 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260928 12:30:47.970425 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:47.966036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.8.13.1:38793)
0928 12:30:47.966603 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:47.966607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:47.966625 (+ 18us) server_negotiation.cc:407] Connection header received
0928 12:30:47.966660 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:47.966661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:47.966675 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:47.966741 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:47.966907 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.967632 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.969033 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:47.969690 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:47.969752 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:47.969831 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:47.970020 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 12:30:47.970187 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":103,"threads_started":1}
I20260928 12:30:47.971055 10114 tablet_service.cc:1467] Processing CreateTablet for tablet 9027aafaf5214a2eace3791ca3634487 (DEFAULT_TABLE table=test-table [id=39db40ea36c44143ae7feb1e2abbee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:47.971065 10394 tablet_service.cc:1467] Processing CreateTablet for tablet 9027aafaf5214a2eace3791ca3634487 (DEFAULT_TABLE table=test-table [id=39db40ea36c44143ae7feb1e2abbee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:47.971063 10254 tablet_service.cc:1467] Processing CreateTablet for tablet 9027aafaf5214a2eace3791ca3634487 (DEFAULT_TABLE table=test-table [id=39db40ea36c44143ae7feb1e2abbee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:47.971424 10114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9027aafaf5214a2eace3791ca3634487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:47.971494 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9027aafaf5214a2eace3791ca3634487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:47.971513 10394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9027aafaf5214a2eace3791ca3634487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:47.975010 10483 tablet_bootstrap.cc:492] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: Bootstrap starting.
I20260928 12:30:47.975660 10483 tablet_bootstrap.cc:654] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:47.975934 10483 log.cc:824] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:47.976039 10485 tablet_bootstrap.cc:492] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: Bootstrap starting.
I20260928 12:30:47.976300 10484 tablet_bootstrap.cc:492] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: Bootstrap starting.
I20260928 12:30:47.976853 10485 tablet_bootstrap.cc:654] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:47.977010 10483 tablet_bootstrap.cc:492] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: No bootstrap required, opened a new log
I20260928 12:30:47.977075 10483 ts_tablet_manager.cc:1397] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:30:47.977147 10485 log.cc:824] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:47.977274 10484 tablet_bootstrap.cc:654] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:47.977603 10484 log.cc:824] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:47.978084 10485 tablet_bootstrap.cc:492] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: No bootstrap required, opened a new log
I20260928 12:30:47.978165 10485 ts_tablet_manager.cc:1397] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:30:47.978574 10484 tablet_bootstrap.cc:492] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: No bootstrap required, opened a new log
I20260928 12:30:47.978634 10484 ts_tablet_manager.cc:1397] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:30:47.979257 10483 raft_consensus.cc:348] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } }
I20260928 12:30:47.979447 10483 raft_consensus.cc:374] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:47.979482 10483 raft_consensus.cc:729] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56b1f0e179e4cef8520a19aff4b4e30, State: Initialized, Role: FOLLOWER
I20260928 12:30:47.979621 10483 consensus_queue.cc:260] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } }
I20260928 12:30:47.979568 10485 raft_consensus.cc:348] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } }
I20260928 12:30:47.979720 10485 raft_consensus.cc:374] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:47.979744 10485 raft_consensus.cc:729] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a465a63a474d0db37dc956aa6788a1, State: Initialized, Role: FOLLOWER
I20260928 12:30:47.979836 10485 consensus_queue.cc:260] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } }
I20260928 12:30:47.979923 10483 ts_tablet_manager.cc:1428] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:30:47.980055 10180 heartbeater.cc:503] Master 127.8.13.62:45735 was elected leader, sending a full tablet report...
I20260928 12:30:47.980079 10485 ts_tablet_manager.cc:1428] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:30:47.980115 10320 heartbeater.cc:503] Master 127.8.13.62:45735 was elected leader, sending a full tablet report...
I20260928 12:30:47.981534 10484 raft_consensus.cc:348] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } }
I20260928 12:30:47.981699 10484 raft_consensus.cc:374] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:47.981730 10484 raft_consensus.cc:729] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae68cbe1e42642e98af1ac534571b116, State: Initialized, Role: FOLLOWER
I20260928 12:30:47.981868 10484 consensus_queue.cc:260] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } }
I20260928 12:30:47.982147 10484 ts_tablet_manager.cc:1428] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 12:30:47.982187 10460 heartbeater.cc:503] Master 127.8.13.62:45735 was elected leader, sending a full tablet report...
W20260928 12:30:48.004153 10321 tablet.cc:2369] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:30:48.098857 10181 tablet.cc:2369] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:30:48.178077 10461 tablet.cc:2369] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:30:48.197196 10489 raft_consensus.cc:482] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:30:48.197341 10489 raft_consensus.cc:504] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } }
I20260928 12:30:48.197762 10489 leader_election.cc:288] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae68cbe1e42642e98af1ac534571b116 (127.8.13.3:32847), 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519)
I20260928 12:30:48.202471 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.198315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:40335 (local address 127.8.13.2:45519)
0928 12:30:48.198366 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:48.198368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:48.198467 (+ 99us) server_negotiation.cc:407] Connection header received
0928 12:30:48.198557 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:48.198557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:48.198567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:48.198604 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:48.198818 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.199564 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.201421 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.202063 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.202144 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.202156 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:48.202349 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 12:30:48.202387 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:30:48.202493 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.198146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:42439 (local address 127.8.13.3:32847)
0928 12:30:48.198197 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:48.198199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:48.198264 (+ 65us) server_negotiation.cc:407] Connection header received
0928 12:30:48.198382 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:48.198383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:48.198396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:48.198445 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:48.198656 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.199579 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.201421 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.202062 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.202144 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.202155 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:48.202349 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 12:30:48.202387 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:30:48.202509 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.198071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:32847 (local address 127.8.13.1:42439)
0928 12:30:48.198189 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:48.198201 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:48.198272 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:48.198493 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:48.198494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:48.198496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:48.198586 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.198588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.199686 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.199687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.201291 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.201295 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.202152 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.202153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.202263 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.202272 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 12:30:48.202340 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:30:48.202397 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":75,"thread_start_us":60,"threads_started":1}
I20260928 12:30:48.202509 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.198241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:45519 (local address 127.8.13.1:40335)
0928 12:30:48.198351 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:48.198361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:48.198493 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:48.198608 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:48.198609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:48.198611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:48.198666 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.198667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.199680 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.199681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.201293 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.201295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.202153 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.202154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.202263 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.202273 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 12:30:48.202340 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 12:30:48.202397 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":107,"thread_start_us":55,"threads_started":1}
I20260928 12:30:48.202916 10414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9027aafaf5214a2eace3791ca3634487" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae68cbe1e42642e98af1ac534571b116" is_pre_election: true
I20260928 12:30:48.202979 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9027aafaf5214a2eace3791ca3634487" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a465a63a474d0db37dc956aa6788a1" is_pre_election: true
I20260928 12:30:48.203081 10414 raft_consensus.cc:2454] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 0.
I20260928 12:30:48.203101 10274 raft_consensus.cc:2454] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 0.
I20260928 12:30:48.203512 10067 leader_election.cc:302] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae68cbe1e42642e98af1ac534571b116, f56b1f0e179e4cef8520a19aff4b4e30; no voters:
I20260928 12:30:48.203655 10489 raft_consensus.cc:2781] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:30:48.203724 10489 raft_consensus.cc:482] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:30:48.203748 10489 raft_consensus.cc:3037] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:48.204484 10489 raft_consensus.cc:504] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } }
I20260928 12:30:48.204618 10489 leader_election.cc:288] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 election: Requested vote from peers ae68cbe1e42642e98af1ac534571b116 (127.8.13.3:32847), 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519)
I20260928 12:30:48.204818 10414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9027aafaf5214a2eace3791ca3634487" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae68cbe1e42642e98af1ac534571b116"
I20260928 12:30:48.204854 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9027aafaf5214a2eace3791ca3634487" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a465a63a474d0db37dc956aa6788a1"
I20260928 12:30:48.204914 10414 raft_consensus.cc:3037] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:48.204926 10274 raft_consensus.cc:3037] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:48.205833 10414 raft_consensus.cc:2454] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 1.
I20260928 12:30:48.205833 10274 raft_consensus.cc:2454] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 1.
I20260928 12:30:48.205996 10067 leader_election.cc:302] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a465a63a474d0db37dc956aa6788a1, f56b1f0e179e4cef8520a19aff4b4e30; no voters:
I20260928 12:30:48.206089 10489 raft_consensus.cc:2781] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:48.206226 10489 raft_consensus.cc:686] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 LEADER]: Becoming Leader. State: Replica: f56b1f0e179e4cef8520a19aff4b4e30, State: Running, Role: LEADER
I20260928 12:30:48.206336 10489 consensus_queue.cc:237] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } }
I20260928 12:30:48.207149 9977 catalog_manager.cc:5495] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 reported cstate change: term changed from 0 to 1, leader changed from <none> to f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1). New cstate: current_term: 1 leader_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } health_report { overall_health: UNKNOWN } } }
Sep 28 12:30:48 dist-test-slave-g31s krb5kdc[9932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598647, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:48.282301 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.274578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.8.13.62:45735)
0928 12:30:48.274638 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:48.274641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:48.274753 (+ 112us) server_negotiation.cc:407] Connection header received
0928 12:30:48.274821 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:48.274822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:48.274832 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:48.274876 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:48.275355 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.275860 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.276537 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.276716 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.276771 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.279202 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:48.279204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:48.279204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:48.279211 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:48.280543 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:48.281023 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:48.281025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:48.281026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:48.281163 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:48.281479 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:48.281480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:48.281480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:48.281819 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:48.281909 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:48.282115 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 12:30:48.282218 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:30:48.282542 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.274526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45735 (local address 127.0.0.1:60366)
0928 12:30:48.274685 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:48.274694 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:48.274762 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:48.274896 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:48.274896 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:48.275218 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:48.275272 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.275276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.275931 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.275932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.276463 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.276465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.276773 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.276774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.276822 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.277369 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:48.277374 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:48.279051 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:48.280638 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:48.280646 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:48.280649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:48.280861 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:48.281312 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:48.281313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:48.281313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:48.281390 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:48.281936 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:48.281946 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:48.282023 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 12:30:48.282085 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:30:48.282461 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":214,"thread_start_us":99,"threads_started":1}
I20260928 12:30:48.285012 9977 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:60366:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:30:48.285276 9977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:30:48.287321 10114 tablet_service.cc:1467] Processing CreateTablet for tablet a0b9a1c8a8cb4d67932a6c3e88cea9a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=828a1aa989dc45ceb8da03ed3f1b97e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:30:48.287369 10394 tablet_service.cc:1467] Processing CreateTablet for tablet a0b9a1c8a8cb4d67932a6c3e88cea9a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=828a1aa989dc45ceb8da03ed3f1b97e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:30:48.287360 10254 tablet_service.cc:1467] Processing CreateTablet for tablet a0b9a1c8a8cb4d67932a6c3e88cea9a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=828a1aa989dc45ceb8da03ed3f1b97e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:30:48.287493 10114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b9a1c8a8cb4d67932a6c3e88cea9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:48.287528 10394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b9a1c8a8cb4d67932a6c3e88cea9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:48.287539 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b9a1c8a8cb4d67932a6c3e88cea9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:48.288700 10483 tablet_bootstrap.cc:492] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: Bootstrap starting.
I20260928 12:30:48.289085 10485 tablet_bootstrap.cc:492] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1: Bootstrap starting.
I20260928 12:30:48.289166 10483 tablet_bootstrap.cc:654] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:48.289176 10484 tablet_bootstrap.cc:492] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116: Bootstrap starting.
I20260928 12:30:48.289420 10485 tablet_bootstrap.cc:654] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:48.289685 10484 tablet_bootstrap.cc:654] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:48.290023 10485 tablet_bootstrap.cc:492] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1: No bootstrap required, opened a new log
I20260928 12:30:48.290069 10485 ts_tablet_manager.cc:1397] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:48.290110 10483 tablet_bootstrap.cc:492] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: No bootstrap required, opened a new log
I20260928 12:30:48.290154 10483 ts_tablet_manager.cc:1397] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:48.290220 10485 raft_consensus.cc:348] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.290302 10485 raft_consensus.cc:374] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:48.290304 10484 tablet_bootstrap.cc:492] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116: No bootstrap required, opened a new log
I20260928 12:30:48.290323 10485 raft_consensus.cc:729] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a465a63a474d0db37dc956aa6788a1, State: Initialized, Role: FOLLOWER
I20260928 12:30:48.290294 10483 raft_consensus.cc:348] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.290340 10484 ts_tablet_manager.cc:1397] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:48.290369 10483 raft_consensus.cc:374] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:48.290388 10483 raft_consensus.cc:729] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56b1f0e179e4cef8520a19aff4b4e30, State: Initialized, Role: FOLLOWER
I20260928 12:30:48.290376 10485 consensus_queue.cc:260] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.290444 10483 consensus_queue.cc:260] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.290506 10485 ts_tablet_manager.cc:1428] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:48.290534 10483 ts_tablet_manager.cc:1428] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:48.290515 10484 raft_consensus.cc:348] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.290575 10484 raft_consensus.cc:374] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:48.290594 10484 raft_consensus.cc:729] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae68cbe1e42642e98af1ac534571b116, State: Initialized, Role: FOLLOWER
I20260928 12:30:48.290575 10489 tablet_replica.cc:439] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } }
I20260928 12:30:48.290638 10489 tablet_replica.cc:442] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:48.290635 10484 consensus_queue.cc:260] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.290638 10490 tablet_replica.cc:439] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } }
I20260928 12:30:48.290714 10484 ts_tablet_manager.cc:1428] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:48.290719 10490 tablet_replica.cc:442] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:48.290840 10491 tablet_replica.cc:439] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } }
I20260928 12:30:48.290920 10491 tablet_replica.cc:442] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:48.418833 10489 raft_consensus.cc:482] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:30:48.418936 10489 raft_consensus.cc:504] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.419090 10489 leader_election.cc:288] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519), ae68cbe1e42642e98af1ac534571b116 (127.8.13.3:32847)
I20260928 12:30:48.419322 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b9a1c8a8cb4d67932a6c3e88cea9a9" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a465a63a474d0db37dc956aa6788a1" is_pre_election: true
I20260928 12:30:48.419355 10414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b9a1c8a8cb4d67932a6c3e88cea9a9" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae68cbe1e42642e98af1ac534571b116" is_pre_election: true
I20260928 12:30:48.419442 10414 raft_consensus.cc:2454] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 0.
I20260928 12:30:48.419442 10274 raft_consensus.cc:2454] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 0.
I20260928 12:30:48.419649 10067 leader_election.cc:302] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a465a63a474d0db37dc956aa6788a1, f56b1f0e179e4cef8520a19aff4b4e30; no voters:
I20260928 12:30:48.419759 10489 raft_consensus.cc:2781] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:30:48.419807 10489 raft_consensus.cc:482] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:30:48.419822 10489 raft_consensus.cc:3037] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:48.420470 10489 raft_consensus.cc:504] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.420591 10489 leader_election.cc:288] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 election: Requested vote from peers 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519), ae68cbe1e42642e98af1ac534571b116 (127.8.13.3:32847)
I20260928 12:30:48.420806 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b9a1c8a8cb4d67932a6c3e88cea9a9" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a465a63a474d0db37dc956aa6788a1"
I20260928 12:30:48.420883 10274 raft_consensus.cc:3037] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:48.420858 10414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b9a1c8a8cb4d67932a6c3e88cea9a9" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae68cbe1e42642e98af1ac534571b116"
I20260928 12:30:48.420954 10414 raft_consensus.cc:3037] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:48.421486 10274 raft_consensus.cc:2454] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 1.
I20260928 12:30:48.421564 10414 raft_consensus.cc:2454] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 1.
I20260928 12:30:48.421651 10067 leader_election.cc:302] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a465a63a474d0db37dc956aa6788a1, f56b1f0e179e4cef8520a19aff4b4e30; no voters:
I20260928 12:30:48.421759 10489 raft_consensus.cc:2781] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:48.421841 10489 raft_consensus.cc:686] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 LEADER]: Becoming Leader. State: Replica: f56b1f0e179e4cef8520a19aff4b4e30, State: Running, Role: LEADER
I20260928 12:30:48.421892 10489 consensus_queue.cc:237] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.422236 10494 tablet_replica.cc:439] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: TxnStatusTablet state changed. Reason: New leader f56b1f0e179e4cef8520a19aff4b4e30. Latest consensus state: current_term: 1 leader_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } }
I20260928 12:30:48.422334 10494 tablet_replica.cc:442] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:30:48.422708 10506 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:30:48.422878 10506 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:30:48.422832 9976 catalog_manager.cc:5495] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 reported cstate change: term changed from 0 to 1, leader changed from <none> to f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1). New cstate: current_term: 1 leader_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } health_report { overall_health: UNKNOWN } } }
I20260928 12:30:48.442096 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.438537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:38793 (local address 127.0.0.1:58290)
0928 12:30:48.438633 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:48.438648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:48.438709 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:48.438883 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:48.438884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:48.438885 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:30:48.438937 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.438940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.439906 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.439909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.440725 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.440729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.441328 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.441330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.441421 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.441442 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:30:48.441860 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:30:48.441886 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 12:30:48.441930 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:30:48.442007 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 12:30:48.442095 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.438659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.8.13.1:38793)
0928 12:30:48.438726 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:48.438728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:48.438740 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:30:48.438782 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:48.438788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:48.438809 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:48.438882 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:48.438998 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.439755 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.440849 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.441187 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.441272 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.441547 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:48.441686 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:48.441908 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:48.441951 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 12:30:48.441981 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:30:48.444427 10274 raft_consensus.cc:1261] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Refusing update from remote peer f56b1f0e179e4cef8520a19aff4b4e30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:48.444413 10414 raft_consensus.cc:1261] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Refusing update from remote peer f56b1f0e179e4cef8520a19aff4b4e30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:48.444787 10489 consensus_queue.cc:1035] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:48.444926 10494 consensus_queue.cc:1035] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:48.446165 10490 tablet_replica.cc:439] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1: TxnStatusTablet state changed. Reason: New leader f56b1f0e179e4cef8520a19aff4b4e30. Latest consensus state: current_term: 1 leader_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } }
I20260928 12:30:48.446267 10490 tablet_replica.cc:442] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:48.446209 10491 tablet_replica.cc:439] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116: TxnStatusTablet state changed. Reason: New leader f56b1f0e179e4cef8520a19aff4b4e30. Latest consensus state: current_term: 1 leader_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } }
I20260928 12:30:48.446312 10491 tablet_replica.cc:442] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:48.447523 10494 tablet_replica.cc:439] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } }
I20260928 12:30:48.447640 10494 tablet_replica.cc:442] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:30:48.447687 10494 tablet_replica.cc:439] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } }
I20260928 12:30:48.447654 10491 tablet_replica.cc:439] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } }
I20260928 12:30:48.447726 10494 tablet_replica.cc:442] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P f56b1f0e179e4cef8520a19aff4b4e30: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:30:48.447741 10491 tablet_replica.cc:442] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P ae68cbe1e42642e98af1ac534571b116: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:48.448678 10490 tablet_replica.cc:439] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } } }
I20260928 12:30:48.448781 10490 tablet_replica.cc:442] T a0b9a1c8a8cb4d67932a6c3e88cea9a9 P 98a465a63a474d0db37dc956aa6788a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:30:48.449496 10511 mvcc.cc:205] Tried to move back new op lower bound from 7334292064025223168 to 7334292063936204800. Current Snapshot: MvccSnapshot[applied={T|T < 7334292064025223168 or (T in {7334292064025223168})}]
I20260928 12:30:48.452009 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.449524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.8.13.1:38793)
0928 12:30:48.449714 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:48.449715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:48.449721 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:30:48.449739 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:48.449739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:48.449746 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:48.449788 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:48.450044 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.450539 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.451289 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.451521 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.451582 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.451770 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:48.451807 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:48.451848 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:48.451900 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 12:30:48.451922 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260928 12:30:48.458181 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.455056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:33331 (local address 127.8.13.1:38793)
0928 12:30:48.455102 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:48.455104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:48.455164 (+ 60us) server_negotiation.cc:407] Connection header received
0928 12:30:48.455239 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:48.455239 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:48.455247 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:48.455291 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:48.455431 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.456008 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.457257 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.457539 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.457619 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.457903 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:48.457943 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:48.457996 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:48.458057 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 12:30:48.458079 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:30:48.458284 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.454937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:38793 (local address 127.8.13.1:33331)
0928 12:30:48.455102 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:48.455114 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:48.455182 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:48.455296 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:48.455297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:48.455308 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:30:48.455377 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.455379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.456087 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.456088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.457094 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.457098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.457644 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.457646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.457714 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.457742 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:30:48.458011 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:30:48.458013 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:30:48.458056 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:30:48.458109 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260928 12:30:48.461427 10414 raft_consensus.cc:1261] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Refusing update from remote peer f56b1f0e179e4cef8520a19aff4b4e30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:48.461354 10274 raft_consensus.cc:1261] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Refusing update from remote peer f56b1f0e179e4cef8520a19aff4b4e30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:48.461655 10489 consensus_queue.cc:1035] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:48.461825 10489 consensus_queue.cc:1035] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:48.463451 10509 mvcc.cc:205] Tried to move back new op lower bound from 7334292064096231424 to 7334292063053676544. Current Snapshot: MvccSnapshot[applied={T|T < 7334292064096231424}]
I20260928 12:30:48.478724 9976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:48.478852 9976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:48.479663 9976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83: Sending DeleteTablet for 3 replicas of tablet 9027aafaf5214a2eace3791ca3634487
I20260928 12:30:48.480003 10394 tablet_service.cc:1514] Processing DeleteTablet for tablet 9027aafaf5214a2eace3791ca3634487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:48 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:53454
I20260928 12:30:48.480058 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet 9027aafaf5214a2eace3791ca3634487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:48 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:58528
I20260928 12:30:48.480268 10526 tablet_replica.cc:330] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: stopping tablet replica
I20260928 12:30:48.480248 10113 tablet_service.cc:1514] Processing DeleteTablet for tablet 9027aafaf5214a2eace3791ca3634487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:48 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260928 12:30:48.480391 10526 raft_consensus.cc:2229] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:48.480458 10526 raft_consensus.cc:2258] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:48.480731 9976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 12:30:48.480835 10526 ts_tablet_manager.cc:1905] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 12:30:48.480907 9976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:30:48.481226 10525 tablet_replica.cc:330] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: stopping tablet replica
I20260928 12:30:48.481350 10525 raft_consensus.cc:2229] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:48.481410 10525 raft_consensus.cc:2258] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:48.481848 10525 ts_tablet_manager.cc:1905] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:48.482393 10394 tablet_service.cc:1467] Processing CreateTablet for tablet bf3874bb78f5484d99eb9c0535b07f5b (DEFAULT_TABLE table=test-table [id=1dc2d44ac2cf42bdb0aaa2a3ff1da8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:48.482499 10394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3874bb78f5484d99eb9c0535b07f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:48.482584 10254 tablet_service.cc:1467] Processing CreateTablet for tablet bf3874bb78f5484d99eb9c0535b07f5b (DEFAULT_TABLE table=test-table [id=1dc2d44ac2cf42bdb0aaa2a3ff1da8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:48.482712 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3874bb78f5484d99eb9c0535b07f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:48.483901 10485 tablet_bootstrap.cc:492] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1: Bootstrap starting.
I20260928 12:30:48.484525 10485 tablet_bootstrap.cc:654] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:48.484687 10113 tablet_service.cc:1467] Processing CreateTablet for tablet bf3874bb78f5484d99eb9c0535b07f5b (DEFAULT_TABLE table=test-table [id=1dc2d44ac2cf42bdb0aaa2a3ff1da8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:48.484833 10113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3874bb78f5484d99eb9c0535b07f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:48.486119 10527 tablet_replica.cc:330] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: stopping tablet replica
I20260928 12:30:48.486285 10484 tablet_bootstrap.cc:492] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116: Bootstrap starting.
I20260928 12:30:48.486585 10526 ts_tablet_manager.cc:1918] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:30:48.486752 10526 log.cc:1196] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/9027aafaf5214a2eace3791ca3634487
I20260928 12:30:48.486800 10527 raft_consensus.cc:2229] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:30:48.486847 10484 tablet_bootstrap.cc:654] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:48.486977 10527 raft_consensus.cc:2258] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:48.487186 10483 tablet_bootstrap.cc:492] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30: Bootstrap starting.
I20260928 12:30:48.487344 10526 ts_tablet_manager.cc:1939] T 9027aafaf5214a2eace3791ca3634487 P 98a465a63a474d0db37dc956aa6788a1: Deleting consensus metadata
I20260928 12:30:48.487525 10527 ts_tablet_manager.cc:1905] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:48.487603 10483 tablet_bootstrap.cc:654] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:48.488157 9963 catalog_manager.cc:4841] TS 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519): tablet 9027aafaf5214a2eace3791ca3634487 (table test-table [id=39db40ea36c44143ae7feb1e2abbee62]) successfully deleted
I20260928 12:30:48.489688 10525 ts_tablet_manager.cc:1918] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:30:48.489789 10525 log.cc:1196] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/9027aafaf5214a2eace3791ca3634487
I20260928 12:30:48.489665 10527 ts_tablet_manager.cc:1918] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:30:48.489910 10527 log.cc:1196] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/wal/wals/9027aafaf5214a2eace3791ca3634487
I20260928 12:30:48.490077 10525 ts_tablet_manager.cc:1939] T 9027aafaf5214a2eace3791ca3634487 P ae68cbe1e42642e98af1ac534571b116: Deleting consensus metadata
I20260928 12:30:48.490243 10527 ts_tablet_manager.cc:1939] T 9027aafaf5214a2eace3791ca3634487 P f56b1f0e179e4cef8520a19aff4b4e30: Deleting consensus metadata
I20260928 12:30:48.490665 9963 catalog_manager.cc:4841] TS ae68cbe1e42642e98af1ac534571b116 (127.8.13.3:32847): tablet 9027aafaf5214a2eace3791ca3634487 (table test-table [id=39db40ea36c44143ae7feb1e2abbee62]) successfully deleted
I20260928 12:30:48.490777 9963 catalog_manager.cc:4841] TS f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1:38793): tablet 9027aafaf5214a2eace3791ca3634487 (table test-table [id=39db40ea36c44143ae7feb1e2abbee62]) successfully deleted
I20260928 12:30:48.491199 10484 tablet_bootstrap.cc:492] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116: No bootstrap required, opened a new log
I20260928 12:30:48.491257 10484 ts_tablet_manager.cc:1397] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 12:30:48.491366 10483 tablet_bootstrap.cc:492] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30: No bootstrap required, opened a new log
I20260928 12:30:48.491415 10483 ts_tablet_manager.cc:1397] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 12:30:48.491425 10484 raft_consensus.cc:348] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.491508 10484 raft_consensus.cc:374] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:48.491525 10484 raft_consensus.cc:729] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae68cbe1e42642e98af1ac534571b116, State: Initialized, Role: FOLLOWER
I20260928 12:30:48.491572 10484 consensus_queue.cc:260] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.491556 10483 raft_consensus.cc:348] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.491641 10483 raft_consensus.cc:374] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:48.491662 10483 raft_consensus.cc:729] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56b1f0e179e4cef8520a19aff4b4e30, State: Initialized, Role: FOLLOWER
I20260928 12:30:48.491673 10484 ts_tablet_manager.cc:1428] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:48.491699 10483 consensus_queue.cc:260] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.491776 10483 ts_tablet_manager.cc:1428] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:48.494280 10485 tablet_bootstrap.cc:492] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1: No bootstrap required, opened a new log
I20260928 12:30:48.494346 10485 ts_tablet_manager.cc:1397] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260928 12:30:48.494513 10485 raft_consensus.cc:348] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.494587 10485 raft_consensus.cc:374] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:48.494601 10485 raft_consensus.cc:729] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a465a63a474d0db37dc956aa6788a1, State: Initialized, Role: FOLLOWER
I20260928 12:30:48.494644 10485 consensus_queue.cc:260] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.494732 10485 ts_tablet_manager.cc:1428] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:48.592384 10491 raft_consensus.cc:482] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:30:48.592494 10491 raft_consensus.cc:504] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.592828 10491 leader_election.cc:288] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1:38793), 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519)
I20260928 12:30:48.596823 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.593123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:37779 (local address 127.8.13.1:38793)
0928 12:30:48.593170 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:48.593172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:48.593330 (+ 158us) server_negotiation.cc:407] Connection header received
0928 12:30:48.593485 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:48.593486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:48.593501 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:48.593563 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:48.593852 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.594327 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.595796 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.596386 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.596467 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.596481 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:48.596684 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 12:30:48.596734 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:30:48.596848 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.593059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:38793 (local address 127.8.13.3:37779)
0928 12:30:48.593249 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:48.593259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:48.593362 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:48.593680 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:48.593681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:48.593682 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:48.593740 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.593741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.594410 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.594412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.595681 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.595685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.596496 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.596497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.596568 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.596574 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:30:48.596646 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 12:30:48.596736 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":25,"thread_start_us":123,"threads_started":1}
I20260928 12:30:48.597524 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.593370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:39227 (local address 127.8.13.2:45519)
0928 12:30:48.593412 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:48.593414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:48.593514 (+ 100us) server_negotiation.cc:407] Connection header received
0928 12:30:48.593582 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:48.593583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:48.593592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:48.593638 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:30:48.593830 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.594539 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.596521 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.597110 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.597176 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.597187 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:48.597404 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 12:30:48.597460 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:30:48.597520 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.593289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:45519 (local address 127.8.13.3:39227)
0928 12:30:48.593460 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:48.593469 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:48.593519 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:48.593652 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:48.593653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:48.593654 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:30:48.593730 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.593732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.594642 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.594643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.595684 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.595686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.597223 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.597224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.597308 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.597312 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:30:48.597386 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 12:30:48.597441 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":56,"threads_started":1}
I20260928 12:30:48.597488 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3874bb78f5484d99eb9c0535b07f5b" candidate_uuid: "ae68cbe1e42642e98af1ac534571b116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" is_pre_election: true
I20260928 12:30:48.597623 10134 raft_consensus.cc:2454] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae68cbe1e42642e98af1ac534571b116 in term 0.
I20260928 12:30:48.597784 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3874bb78f5484d99eb9c0535b07f5b" candidate_uuid: "ae68cbe1e42642e98af1ac534571b116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a465a63a474d0db37dc956aa6788a1" is_pre_election: true
I20260928 12:30:48.597898 10274 raft_consensus.cc:2454] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae68cbe1e42642e98af1ac534571b116 in term 0.
I20260928 12:30:48.598074 10347 leader_election.cc:302] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a465a63a474d0db37dc956aa6788a1, ae68cbe1e42642e98af1ac534571b116; no voters:
I20260928 12:30:48.598265 10491 raft_consensus.cc:2781] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:30:48.598325 10491 raft_consensus.cc:482] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:30:48.598345 10491 raft_consensus.cc:3037] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:48.599074 10491 raft_consensus.cc:504] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.599201 10491 leader_election.cc:288] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [CANDIDATE]: Term 1 election: Requested vote from peers f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1:38793), 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519)
I20260928 12:30:48.599421 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3874bb78f5484d99eb9c0535b07f5b" candidate_uuid: "ae68cbe1e42642e98af1ac534571b116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56b1f0e179e4cef8520a19aff4b4e30"
I20260928 12:30:48.599504 10134 raft_consensus.cc:3037] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:48.599591 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3874bb78f5484d99eb9c0535b07f5b" candidate_uuid: "ae68cbe1e42642e98af1ac534571b116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a465a63a474d0db37dc956aa6788a1"
I20260928 12:30:48.599656 10274 raft_consensus.cc:3037] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:48.600323 10134 raft_consensus.cc:2454] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae68cbe1e42642e98af1ac534571b116 in term 1.
I20260928 12:30:48.600369 10274 raft_consensus.cc:2454] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae68cbe1e42642e98af1ac534571b116 in term 1.
I20260928 12:30:48.600510 10347 leader_election.cc:302] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a465a63a474d0db37dc956aa6788a1, ae68cbe1e42642e98af1ac534571b116; no voters:
I20260928 12:30:48.600703 10491 raft_consensus.cc:2781] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:48.600852 10491 raft_consensus.cc:686] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 1 LEADER]: Becoming Leader. State: Replica: ae68cbe1e42642e98af1ac534571b116, State: Running, Role: LEADER
I20260928 12:30:48.600934 10491 consensus_queue.cc:237] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:48.601696 9975 catalog_manager.cc:5495] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae68cbe1e42642e98af1ac534571b116 (127.8.13.3). New cstate: current_term: 1 leader_uuid: "ae68cbe1e42642e98af1ac534571b116" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } health_report { overall_health: HEALTHY } } }
I20260928 12:30:48.636502 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.633866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.8.13.3:32847)
0928 12:30:48.633925 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:48.633927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:48.633935 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:30:48.633967 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:48.633968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:48.633987 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:48.634050 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:48.634187 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.634665 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.635767 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.635978 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.636041 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.636195 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:48.636306 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:48.636359 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:48.636409 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 12:30:48.636431 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:30:48.642076 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.639480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:38793 (local address 127.8.13.3:51319)
0928 12:30:48.639668 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:48.639678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:48.639742 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:48.639938 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:48.639939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:48.639942 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:30:48.640040 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.640041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.640664 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.640666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.641255 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.641257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.641625 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.641626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.641679 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.641698 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:30:48.641881 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:30:48.641883 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:30:48.641922 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 12:30:48.641982 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260928 12:30:48.642048 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.639551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:51319 (local address 127.8.13.1:38793)
0928 12:30:48.639604 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:48.639605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:48.639737 (+ 132us) server_negotiation.cc:407] Connection header received
0928 12:30:48.639812 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:48.639813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:48.639823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:48.639938 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:48.640107 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.640593 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.641334 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.641538 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.641596 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.641772 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:48.641830 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:48.641880 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:48.641936 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 12:30:48.641957 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:30:48.644963 10274 raft_consensus.cc:1261] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Refusing update from remote peer ae68cbe1e42642e98af1ac534571b116: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:48.644985 10134 raft_consensus.cc:1261] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Refusing update from remote peer ae68cbe1e42642e98af1ac534571b116: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:48.645263 10491 consensus_queue.cc:1035] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:48.645444 10491 consensus_queue.cc:1035] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:48.646914 10510 mvcc.cc:205] Tried to move back new op lower bound from 7334292064847659008 to 7334292064669945856. Current Snapshot: MvccSnapshot[applied={T|T < 7334292064847659008}]
I20260928 12:30:48.654819 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.651508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:32847 (local address 127.8.13.1:45723)
0928 12:30:48.651606 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:48.651620 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:48.651662 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:48.651880 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:48.651882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:48.651883 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:30:48.651953 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.651955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.652952 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.652954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.653867 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:48.653869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.654339 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.654340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:48.654389 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.654396 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:30:48.654652 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:30:48.654655 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:30:48.654705 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:30:48.654753 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 12:30:48.654829 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:48.651730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:45723 (local address 127.8.13.3:32847)
0928 12:30:48.651775 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:48.651777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:48.651785 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:30:48.651805 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:48.651806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:48.651820 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:48.651864 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:48.652045 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.652822 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.653948 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:48.654242 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:48.654322 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:48.654497 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:48.654536 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:48.654597 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:48.654722 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 12:30:48.654750 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 12:30:48.655195 10511 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 12:30:50.183727 9976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:50.183857 9976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:50.185024 9976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83: Sending DeleteTablet for 3 replicas of tablet bf3874bb78f5484d99eb9c0535b07f5b
I20260928 12:30:50.185330 10113 tablet_service.cc:1514] Processing DeleteTablet for tablet bf3874bb78f5484d99eb9c0535b07f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:50 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260928 12:30:50.185384 10394 tablet_service.cc:1514] Processing DeleteTablet for tablet bf3874bb78f5484d99eb9c0535b07f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:50 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:53454
I20260928 12:30:50.185606 10564 tablet_replica.cc:330] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116: stopping tablet replica
I20260928 12:30:50.185669 10564 raft_consensus.cc:2229] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:30:50.185770 10564 raft_consensus.cc:2258] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:50.186138 10564 ts_tablet_manager.cc:1905] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:50.186532 10563 tablet_replica.cc:330] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30: stopping tablet replica
I20260928 12:30:50.186647 10563 raft_consensus.cc:2229] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:50.186753 10563 raft_consensus.cc:2258] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:50.187054 10563 ts_tablet_manager.cc:1905] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:50.188930 10564 ts_tablet_manager.cc:1918] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:30:50.189002 10564 log.cc:1196] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/bf3874bb78f5484d99eb9c0535b07f5b
I20260928 12:30:50.189239 10564 ts_tablet_manager.cc:1939] T bf3874bb78f5484d99eb9c0535b07f5b P ae68cbe1e42642e98af1ac534571b116: Deleting consensus metadata
I20260928 12:30:50.189662 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet bf3874bb78f5484d99eb9c0535b07f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:50 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:58528
I20260928 12:30:50.189853 9963 catalog_manager.cc:4841] TS ae68cbe1e42642e98af1ac534571b116 (127.8.13.3:32847): tablet bf3874bb78f5484d99eb9c0535b07f5b (table test-table [id=1dc2d44ac2cf42bdb0aaa2a3ff1da8cc]) successfully deleted
I20260928 12:30:50.190820 10563 ts_tablet_manager.cc:1918] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:30:50.190882 10563 log.cc:1196] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/wal/wals/bf3874bb78f5484d99eb9c0535b07f5b
I20260928 12:30:50.191073 10563 ts_tablet_manager.cc:1939] T bf3874bb78f5484d99eb9c0535b07f5b P f56b1f0e179e4cef8520a19aff4b4e30: Deleting consensus metadata
I20260928 12:30:50.191265 10566 tablet_replica.cc:330] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1: stopping tablet replica
I20260928 12:30:50.191331 10566 raft_consensus.cc:2229] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:50.191370 10566 raft_consensus.cc:2258] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:50.191546 9963 catalog_manager.cc:4841] TS f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1:38793): tablet bf3874bb78f5484d99eb9c0535b07f5b (table test-table [id=1dc2d44ac2cf42bdb0aaa2a3ff1da8cc]) successfully deleted
I20260928 12:30:50.191608 10566 ts_tablet_manager.cc:1905] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:50.193109 10566 ts_tablet_manager.cc:1918] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:30:50.193167 10566 log.cc:1196] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/bf3874bb78f5484d99eb9c0535b07f5b
I20260928 12:30:50.193343 10566 ts_tablet_manager.cc:1939] T bf3874bb78f5484d99eb9c0535b07f5b P 98a465a63a474d0db37dc956aa6788a1: Deleting consensus metadata
I20260928 12:30:50.193738 9963 catalog_manager.cc:4841] TS 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519): tablet bf3874bb78f5484d99eb9c0535b07f5b (table test-table [id=1dc2d44ac2cf42bdb0aaa2a3ff1da8cc]) successfully deleted
Sep 28 12:30:50 dist-test-slave-g31s krb5kdc[9932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598650, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:30:50 dist-test-slave-g31s krb5kdc[9932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598650, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:50.213450 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:50.205973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.8.13.62:45735)
0928 12:30:50.206143 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:50.206145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:50.206155 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:30:50.206308 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:50.206311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:50.206333 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:50.206458 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:50.207202 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:50.207744 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:50.208527 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:50.208737 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:50.208800 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:50.211177 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:50.211179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:50.211179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:50.211186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:50.212533 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:50.212865 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:50.212865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:50.212866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:50.212914 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:50.213032 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:50.213032 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:50.213032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:50.213146 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:50.213216 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:50.213295 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 12:30:50.213364 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260928 12:30:50.215011 9976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:30:50.215160 9976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:30:50.216601 10113 tablet_service.cc:1467] Processing CreateTablet for tablet acabb2ef877f412089d637132aa05f5d (DEFAULT_TABLE table=test-table [id=d527a4d4b4fe45bc9e5ab69c19ed8ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:50.216611 10254 tablet_service.cc:1467] Processing CreateTablet for tablet acabb2ef877f412089d637132aa05f5d (DEFAULT_TABLE table=test-table [id=d527a4d4b4fe45bc9e5ab69c19ed8ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:50.216629 10394 tablet_service.cc:1467] Processing CreateTablet for tablet acabb2ef877f412089d637132aa05f5d (DEFAULT_TABLE table=test-table [id=d527a4d4b4fe45bc9e5ab69c19ed8ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:50.216754 10394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acabb2ef877f412089d637132aa05f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:50.216755 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acabb2ef877f412089d637132aa05f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:50.216755 10113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acabb2ef877f412089d637132aa05f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:50.218321 10575 tablet_bootstrap.cc:492] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1: Bootstrap starting.
I20260928 12:30:50.218308 10576 tablet_bootstrap.cc:492] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116: Bootstrap starting.
I20260928 12:30:50.218678 10576 tablet_bootstrap.cc:654] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:50.218678 10575 tablet_bootstrap.cc:654] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:50.218988 10574 tablet_bootstrap.cc:492] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30: Bootstrap starting.
I20260928 12:30:50.219277 10575 tablet_bootstrap.cc:492] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1: No bootstrap required, opened a new log
I20260928 12:30:50.219321 10575 ts_tablet_manager.cc:1397] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:50.219341 10574 tablet_bootstrap.cc:654] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:50.219353 10576 tablet_bootstrap.cc:492] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116: No bootstrap required, opened a new log
I20260928 12:30:50.219394 10576 ts_tablet_manager.cc:1397] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:50.219455 10575 raft_consensus.cc:348] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.219537 10575 raft_consensus.cc:374] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:50.219558 10575 raft_consensus.cc:729] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a465a63a474d0db37dc956aa6788a1, State: Initialized, Role: FOLLOWER
I20260928 12:30:50.219512 10576 raft_consensus.cc:348] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.219580 10576 raft_consensus.cc:374] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:50.219600 10576 raft_consensus.cc:729] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae68cbe1e42642e98af1ac534571b116, State: Initialized, Role: FOLLOWER
I20260928 12:30:50.219606 10575 consensus_queue.cc:260] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.219656 10576 consensus_queue.cc:260] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.219769 10575 ts_tablet_manager.cc:1428] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:50.219921 10576 ts_tablet_manager.cc:1428] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:50.220510 10574 tablet_bootstrap.cc:492] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30: No bootstrap required, opened a new log
I20260928 12:30:50.220556 10574 ts_tablet_manager.cc:1397] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:30:50.220705 10574 raft_consensus.cc:348] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.220768 10574 raft_consensus.cc:374] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:50.220780 10574 raft_consensus.cc:729] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56b1f0e179e4cef8520a19aff4b4e30, State: Initialized, Role: FOLLOWER
I20260928 12:30:50.220827 10574 consensus_queue.cc:260] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.220923 10574 ts_tablet_manager.cc:1428] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:50.512856 10555 raft_consensus.cc:482] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:30:50.512976 10555 raft_consensus.cc:504] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.513141 10555 leader_election.cc:288] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1:38793), 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519)
I20260928 12:30:50.513403 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acabb2ef877f412089d637132aa05f5d" candidate_uuid: "ae68cbe1e42642e98af1ac534571b116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a465a63a474d0db37dc956aa6788a1" is_pre_election: true
I20260928 12:30:50.513408 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acabb2ef877f412089d637132aa05f5d" candidate_uuid: "ae68cbe1e42642e98af1ac534571b116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" is_pre_election: true
I20260928 12:30:50.513494 10274 raft_consensus.cc:2454] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae68cbe1e42642e98af1ac534571b116 in term 0.
I20260928 12:30:50.513512 10134 raft_consensus.cc:2454] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae68cbe1e42642e98af1ac534571b116 in term 0.
I20260928 12:30:50.513640 10347 leader_election.cc:302] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a465a63a474d0db37dc956aa6788a1, ae68cbe1e42642e98af1ac534571b116; no voters:
I20260928 12:30:50.513778 10555 raft_consensus.cc:2781] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:30:50.513860 10555 raft_consensus.cc:482] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:30:50.513886 10555 raft_consensus.cc:3037] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:50.514747 10555 raft_consensus.cc:504] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.514860 10555 leader_election.cc:288] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [CANDIDATE]: Term 1 election: Requested vote from peers f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1:38793), 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519)
I20260928 12:30:50.515057 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acabb2ef877f412089d637132aa05f5d" candidate_uuid: "ae68cbe1e42642e98af1ac534571b116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56b1f0e179e4cef8520a19aff4b4e30"
I20260928 12:30:50.515086 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acabb2ef877f412089d637132aa05f5d" candidate_uuid: "ae68cbe1e42642e98af1ac534571b116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a465a63a474d0db37dc956aa6788a1"
I20260928 12:30:50.515139 10134 raft_consensus.cc:3037] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:50.515154 10274 raft_consensus.cc:3037] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:50.515723 10274 raft_consensus.cc:2454] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae68cbe1e42642e98af1ac534571b116 in term 1.
I20260928 12:30:50.515764 10134 raft_consensus.cc:2454] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae68cbe1e42642e98af1ac534571b116 in term 1.
I20260928 12:30:50.515901 10347 leader_election.cc:302] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a465a63a474d0db37dc956aa6788a1, ae68cbe1e42642e98af1ac534571b116; no voters:
I20260928 12:30:50.515985 10555 raft_consensus.cc:2781] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:50.516043 10555 raft_consensus.cc:686] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 1 LEADER]: Becoming Leader. State: Replica: ae68cbe1e42642e98af1ac534571b116, State: Running, Role: LEADER
I20260928 12:30:50.516095 10555 consensus_queue.cc:237] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.516677 9976 catalog_manager.cc:5495] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae68cbe1e42642e98af1ac534571b116 (127.8.13.3). New cstate: current_term: 1 leader_uuid: "ae68cbe1e42642e98af1ac534571b116" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } health_report { overall_health: HEALTHY } } }
I20260928 12:30:50.578930 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:50.576550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.8.13.3:32847)
0928 12:30:50.576748 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:50.576749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:50.576756 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:30:50.576775 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:50.576775 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:50.576785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:50.576868 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:50.576997 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:50.577504 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:50.578286 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:50.578497 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:50.578553 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:50.578676 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:50.578713 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:50.578753 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:50.578812 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 12:30:50.578834 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260928 12:30:50.581956 10134 raft_consensus.cc:1261] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Refusing update from remote peer ae68cbe1e42642e98af1ac534571b116: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:50.582109 10274 raft_consensus.cc:1261] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Refusing update from remote peer ae68cbe1e42642e98af1ac534571b116: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:50.582198 10555 consensus_queue.cc:1035] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:50.582574 10555 consensus_queue.cc:1035] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:50.592801 9977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:50.592905 9977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:50.593547 9977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83: Sending DeleteTablet for 3 replicas of tablet acabb2ef877f412089d637132aa05f5d
I20260928 12:30:50.593791 10113 tablet_service.cc:1514] Processing DeleteTablet for tablet acabb2ef877f412089d637132aa05f5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:50 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260928 12:30:50.593860 10394 tablet_service.cc:1514] Processing DeleteTablet for tablet acabb2ef877f412089d637132aa05f5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:50 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:53454
I20260928 12:30:50.593922 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet acabb2ef877f412089d637132aa05f5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:50 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:58528
I20260928 12:30:50.594022 10563 tablet_replica.cc:330] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30: stopping tablet replica
I20260928 12:30:50.594084 10563 raft_consensus.cc:2229] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:50.594053 9977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 12:30:50.594132 10563 raft_consensus.cc:2258] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 12:30:50.594251 9977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:30:50.594264 10564 tablet_replica.cc:330] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116: stopping tablet replica
I20260928 12:30:50.594327 10564 raft_consensus.cc:2229] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:30:50.594388 10564 raft_consensus.cc:2258] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:50.594476 10566 tablet_replica.cc:330] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1: stopping tablet replica
I20260928 12:30:50.594530 10566 raft_consensus.cc:2229] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:50.594583 10566 raft_consensus.cc:2258] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:50.594710 10564 ts_tablet_manager.cc:1905] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:50.594888 10566 ts_tablet_manager.cc:1905] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:50.595256 10563 ts_tablet_manager.cc:1905] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:50.596686 10566 ts_tablet_manager.cc:1918] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:30:50.596748 10566 log.cc:1196] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/acabb2ef877f412089d637132aa05f5d
I20260928 12:30:50.596766 10563 ts_tablet_manager.cc:1918] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:30:50.596822 10563 log.cc:1196] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-0/wal/wals/acabb2ef877f412089d637132aa05f5d
I20260928 12:30:50.596962 10564 ts_tablet_manager.cc:1918] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:30:50.596992 10566 ts_tablet_manager.cc:1939] T acabb2ef877f412089d637132aa05f5d P 98a465a63a474d0db37dc956aa6788a1: Deleting consensus metadata
I20260928 12:30:50.597021 10564 log.cc:1196] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/acabb2ef877f412089d637132aa05f5d
I20260928 12:30:50.597023 10563 ts_tablet_manager.cc:1939] T acabb2ef877f412089d637132aa05f5d P f56b1f0e179e4cef8520a19aff4b4e30: Deleting consensus metadata
I20260928 12:30:50.597203 10564 ts_tablet_manager.cc:1939] T acabb2ef877f412089d637132aa05f5d P ae68cbe1e42642e98af1ac534571b116: Deleting consensus metadata
I20260928 12:30:50.597350 9963 catalog_manager.cc:4841] TS 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519): tablet acabb2ef877f412089d637132aa05f5d (table test-table [id=d527a4d4b4fe45bc9e5ab69c19ed8ecf]) successfully deleted
I20260928 12:30:50.597424 10113 tablet_service.cc:1467] Processing CreateTablet for tablet fec2601f57234bff963bc7a6ba1db341 (DEFAULT_TABLE table=test-table [id=22ae598fe40742bdb8da24927c67d01a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:50.597491 9963 catalog_manager.cc:4841] TS ae68cbe1e42642e98af1ac534571b116 (127.8.13.3:32847): tablet acabb2ef877f412089d637132aa05f5d (table test-table [id=d527a4d4b4fe45bc9e5ab69c19ed8ecf]) successfully deleted
I20260928 12:30:50.597532 10113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec2601f57234bff963bc7a6ba1db341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:50.597571 10394 tablet_service.cc:1467] Processing CreateTablet for tablet fec2601f57234bff963bc7a6ba1db341 (DEFAULT_TABLE table=test-table [id=22ae598fe40742bdb8da24927c67d01a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:50.597669 10394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec2601f57234bff963bc7a6ba1db341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:50.598608 10574 tablet_bootstrap.cc:492] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30: Bootstrap starting.
I20260928 12:30:50.599000 10576 tablet_bootstrap.cc:492] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116: Bootstrap starting.
I20260928 12:30:50.599283 10574 tablet_bootstrap.cc:654] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:50.599623 10576 tablet_bootstrap.cc:654] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:50.599962 10254 tablet_service.cc:1467] Processing CreateTablet for tablet fec2601f57234bff963bc7a6ba1db341 (DEFAULT_TABLE table=test-table [id=22ae598fe40742bdb8da24927c67d01a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:30:50.600106 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec2601f57234bff963bc7a6ba1db341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:50.600332 10576 tablet_bootstrap.cc:492] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116: No bootstrap required, opened a new log
I20260928 12:30:50.600455 10576 ts_tablet_manager.cc:1397] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:50.600662 10576 raft_consensus.cc:348] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.600733 10576 raft_consensus.cc:374] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:50.600749 10576 raft_consensus.cc:729] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae68cbe1e42642e98af1ac534571b116, State: Initialized, Role: FOLLOWER
I20260928 12:30:50.600805 10576 consensus_queue.cc:260] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.600906 10576 ts_tablet_manager.cc:1428] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:50.601822 10575 tablet_bootstrap.cc:492] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1: Bootstrap starting.
I20260928 12:30:50.601970 9963 catalog_manager.cc:4841] TS f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1:38793): tablet acabb2ef877f412089d637132aa05f5d (table test-table [id=d527a4d4b4fe45bc9e5ab69c19ed8ecf]) successfully deleted
I20260928 12:30:50.602818 10574 tablet_bootstrap.cc:492] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30: No bootstrap required, opened a new log
I20260928 12:30:50.602895 10574 ts_tablet_manager.cc:1397] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 12:30:50.603026 10575 tablet_bootstrap.cc:654] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:50.603093 10574 raft_consensus.cc:348] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.603194 10574 raft_consensus.cc:374] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:50.603206 10574 raft_consensus.cc:729] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56b1f0e179e4cef8520a19aff4b4e30, State: Initialized, Role: FOLLOWER
I20260928 12:30:50.603255 10574 consensus_queue.cc:260] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.603395 10574 ts_tablet_manager.cc:1428] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:50.604560 10575 tablet_bootstrap.cc:492] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1: No bootstrap required, opened a new log
I20260928 12:30:50.604635 10575 ts_tablet_manager.cc:1397] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:30:50.604974 10575 raft_consensus.cc:348] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.606083 10575 raft_consensus.cc:374] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:50.606235 10575 raft_consensus.cc:729] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a465a63a474d0db37dc956aa6788a1, State: Initialized, Role: FOLLOWER
I20260928 12:30:50.606343 10575 consensus_queue.cc:260] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.606475 10575 ts_tablet_manager.cc:1428] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:30:50.780333 10489 raft_consensus.cc:482] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:30:50.780457 10489 raft_consensus.cc:504] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.780673 10489 leader_election.cc:288] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519), ae68cbe1e42642e98af1ac534571b116 (127.8.13.3:32847)
I20260928 12:30:50.781037 10414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec2601f57234bff963bc7a6ba1db341" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae68cbe1e42642e98af1ac534571b116" is_pre_election: true
I20260928 12:30:50.781142 10414 raft_consensus.cc:2454] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 0.
I20260928 12:30:50.781051 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec2601f57234bff963bc7a6ba1db341" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a465a63a474d0db37dc956aa6788a1" is_pre_election: true
I20260928 12:30:50.781231 10274 raft_consensus.cc:2454] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 0.
I20260928 12:30:50.781324 10067 leader_election.cc:302] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae68cbe1e42642e98af1ac534571b116, f56b1f0e179e4cef8520a19aff4b4e30; no voters:
I20260928 12:30:50.781462 10489 raft_consensus.cc:2781] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:30:50.781533 10489 raft_consensus.cc:482] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:30:50.781553 10489 raft_consensus.cc:3037] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:50.782305 10489 raft_consensus.cc:504] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.782454 10489 leader_election.cc:288] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 election: Requested vote from peers 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519), ae68cbe1e42642e98af1ac534571b116 (127.8.13.3:32847)
I20260928 12:30:50.782722 10414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec2601f57234bff963bc7a6ba1db341" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae68cbe1e42642e98af1ac534571b116"
I20260928 12:30:50.782733 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec2601f57234bff963bc7a6ba1db341" candidate_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a465a63a474d0db37dc956aa6788a1"
I20260928 12:30:50.782809 10414 raft_consensus.cc:3037] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:50.782822 10274 raft_consensus.cc:3037] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:50.783396 10274 raft_consensus.cc:2454] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 1.
I20260928 12:30:50.783591 10067 leader_election.cc:302] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a465a63a474d0db37dc956aa6788a1, f56b1f0e179e4cef8520a19aff4b4e30; no voters:
I20260928 12:30:50.783651 10414 raft_consensus.cc:2454] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56b1f0e179e4cef8520a19aff4b4e30 in term 1.
I20260928 12:30:50.783692 10489 raft_consensus.cc:2781] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:50.783753 10489 raft_consensus.cc:686] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [term 1 LEADER]: Becoming Leader. State: Replica: f56b1f0e179e4cef8520a19aff4b4e30, State: Running, Role: LEADER
I20260928 12:30:50.783800 10489 consensus_queue.cc:237] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } }
I20260928 12:30:50.784453 9977 catalog_manager.cc:5495] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 reported cstate change: term changed from 0 to 1, leader changed from <none> to f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1). New cstate: current_term: 1 leader_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56b1f0e179e4cef8520a19aff4b4e30" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 } health_report { overall_health: UNKNOWN } } }
I20260928 12:30:50.831851 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:50.829437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.8.13.1:38793)
0928 12:30:50.829624 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:50.829625 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:50.829631 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:30:50.829648 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:50.829648 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:50.829657 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:50.829715 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:50.829867 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:50.830466 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:50.831212 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:50.831422 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:50.831475 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:50.831609 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:50.831643 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:50.831686 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:50.831738 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 12:30:50.831759 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
I20260928 12:30:50.834169 10414 raft_consensus.cc:1261] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Refusing update from remote peer f56b1f0e179e4cef8520a19aff4b4e30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:50.834169 10274 raft_consensus.cc:1261] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Refusing update from remote peer f56b1f0e179e4cef8520a19aff4b4e30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:50.834399 10545 consensus_queue.cc:1035] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae68cbe1e42642e98af1ac534571b116" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 32847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:50.834506 10489 consensus_queue.cc:1035] T fec2601f57234bff963bc7a6ba1db341 P f56b1f0e179e4cef8520a19aff4b4e30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a465a63a474d0db37dc956aa6788a1" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:30:50.838889 10543 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 12:30:52.357750 9977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:52.357898 9977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:30:52.359258 9977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b07aeb9ad324e1193721f3a92b03d83: Sending DeleteTablet for 3 replicas of tablet fec2601f57234bff963bc7a6ba1db341
I20260928 12:30:52.359632 10114 tablet_service.cc:1514] Processing DeleteTablet for tablet fec2601f57234bff963bc7a6ba1db341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:52 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260928 12:30:52.359654 10394 tablet_service.cc:1514] Processing DeleteTablet for tablet fec2601f57234bff963bc7a6ba1db341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:52 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:53454
I20260928 12:30:52.359906 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 10049
I20260928 12:30:52.359967 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet fec2601f57234bff963bc7a6ba1db341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:30:52 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:58528
I20260928 12:30:52.360530 10617 tablet_replica.cc:330] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1: stopping tablet replica
I20260928 12:30:52.360585 10617 raft_consensus.cc:2229] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:52.360625 10617 raft_consensus.cc:2258] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:52.361119 10617 ts_tablet_manager.cc:1905] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:30:52.362916 10617 ts_tablet_manager.cc:1918] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:30:52.362974 10617 log.cc:1196] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/fec2601f57234bff963bc7a6ba1db341
I20260928 12:30:52.363166 10617 ts_tablet_manager.cc:1939] T fec2601f57234bff963bc7a6ba1db341 P 98a465a63a474d0db37dc956aa6788a1: Deleting consensus metadata
I20260928 12:30:52.363596 9963 catalog_manager.cc:4841] TS 98a465a63a474d0db37dc956aa6788a1 (127.8.13.2:45519): tablet fec2601f57234bff963bc7a6ba1db341 (table test-table [id=22ae598fe40742bdb8da24927c67d01a]) successfully deleted
I20260928 12:30:52.365938 10615 tablet_replica.cc:330] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116: stopping tablet replica
I20260928 12:30:52.366035 10615 raft_consensus.cc:2229] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:30:52.366078 10615 raft_consensus.cc:2258] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:30:52.366369 10615 ts_tablet_manager.cc:1905] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 12:30:52.367094 9963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:38793 (error 108)
I20260928 12:30:52.367326 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 10189
I20260928 12:30:52.368659 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:52.368363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:38793 (local address 127.0.0.1:34584)
0928 12:30:52.368549 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:52.368605 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":115,"threads_started":1}
W20260928 12:30:52.368834 9963 catalog_manager.cc:4584] TS f56b1f0e179e4cef8520a19aff4b4e30 (127.8.13.1:38793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fec2601f57234bff963bc7a6ba1db341: Network error: Client connection negotiation failed: client connection to 127.8.13.1:38793: connect: Connection refused (error 111)
I20260928 12:30:52.369088 10615 ts_tablet_manager.cc:1918] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:30:52.369144 10615 log.cc:1196] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/fec2601f57234bff963bc7a6ba1db341
I20260928 12:30:52.369318 10615 ts_tablet_manager.cc:1939] T fec2601f57234bff963bc7a6ba1db341 P ae68cbe1e42642e98af1ac534571b116: Deleting consensus metadata
I20260928 12:30:52.369761 9963 catalog_manager.cc:4841] TS ae68cbe1e42642e98af1ac534571b116 (127.8.13.3:32847): tablet fec2601f57234bff963bc7a6ba1db341 (table test-table [id=22ae598fe40742bdb8da24927c67d01a]) successfully deleted
I20260928 12:30:52.373296 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 10329
I20260928 12:30:52.380036 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 9945
2026-09-28T12:30:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6209 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:52 dist-test-slave-g31s krb5kdc[10622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:52 dist-test-slave-g31s krb5kdc[10622](info): set up 2 sockets
Sep 28 12:30:52 dist-test-slave-g31s krb5kdc[10622](info): commencing operation
krb5kdc: starting...
W20260928 12:30:53.341465 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:30:53 dist-test-slave-g31s krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:30:53.461416 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33151
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:33151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:53.548300 10635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:53.548573 10635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:53.548612 10635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:53.549942 10635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:53.549988 10635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:53.550006 10635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:53.550024 10635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:53.550038 10635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:53.551642 10635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:33151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:53.551918 10635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:53.552143 10635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:30:53.557243 10635 server_base.cc:1034] running on GCE node
W20260928 12:30:53.557252 10641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:53.557209 10643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:53.557209 10640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:53.557821 10635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:53.558149 10635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:53.559329 10635 hybrid_clock.cc:648] HybridClock initialized: now 1790598653559316 us; error 30 us; skew 500 ppm
Sep 28 12:30:53 dist-test-slave-g31s krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598653, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:53.561975 10635 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:30:53.562428 10635 webserver.cc:460] Webserver started at http://127.8.13.62:34579/ using document root <none> and password file <none>
I20260928 12:30:53.562635 10635 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:53.562681 10635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:53.562806 10635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:53.564052 10635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "69eb8157527f4485a5cda8a996f6804b"
format_stamp: "Formatted at 2026-09-28 12:30:53 on dist-test-slave-g31s"
server_key: "2a973c849a06e9b7d613736098061615"
server_key_iv: "797d518f8dde5e9105178924bb97d191"
server_key_version: "encryptionkey@0"
I20260928 12:30:53.564477 10635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "69eb8157527f4485a5cda8a996f6804b"
format_stamp: "Formatted at 2026-09-28 12:30:53 on dist-test-slave-g31s"
server_key: "2a973c849a06e9b7d613736098061615"
server_key_iv: "797d518f8dde5e9105178924bb97d191"
server_key_version: "encryptionkey@0"
I20260928 12:30:53.566030 10635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:53.566845 10650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:53.567149 10635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:53.567260 10635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "69eb8157527f4485a5cda8a996f6804b"
format_stamp: "Formatted at 2026-09-28 12:30:53 on dist-test-slave-g31s"
server_key: "2a973c849a06e9b7d613736098061615"
server_key_iv: "797d518f8dde5e9105178924bb97d191"
server_key_version: "encryptionkey@0"
I20260928 12:30:53.567353 10635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:53.578020 10635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:53.580418 10635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:53.580570 10635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:53.584447 10635 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:33151
I20260928 12:30:53.584496 10702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:33151 every 8 connection(s)
I20260928 12:30:53.584874 10635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:53.585908 10703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:53.586503 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 10635
I20260928 12:30:53.586603 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:30:53.586771 8244 external_mini_cluster.cc:1428] Setting key 00bd16aeb02cc39dfc39594ab22c3c3f
I20260928 12:30:53.588945 10703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b: Bootstrap starting.
I20260928 12:30:53.589681 10703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:53.590018 10703 log.cc:824] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:53.590730 10703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b: No bootstrap required, opened a new log
I20260928 12:30:53.592959 10703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb8157527f4485a5cda8a996f6804b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33151 } }
I20260928 12:30:53.593106 10703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:53.593139 10703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69eb8157527f4485a5cda8a996f6804b, State: Initialized, Role: FOLLOWER
I20260928 12:30:53.593266 10703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb8157527f4485a5cda8a996f6804b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33151 } }
I20260928 12:30:53.593339 10703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:53.593370 10703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:53.593427 10703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [term 0 FOLLOWER]: Advancing to term 1
Sep 28 12:30:53 dist-test-slave-g31s krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:53.594066 10703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb8157527f4485a5cda8a996f6804b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33151 } }
I20260928 12:30:53.594167 10703 leader_election.cc:302] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69eb8157527f4485a5cda8a996f6804b; no voters:
I20260928 12:30:53.594357 10703 leader_election.cc:288] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:53.594435 10708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:53.594584 10708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [term 1 LEADER]: Becoming Leader. State: Replica: 69eb8157527f4485a5cda8a996f6804b, State: Running, Role: LEADER
I20260928 12:30:53.594640 10703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:53.594686 10708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb8157527f4485a5cda8a996f6804b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33151 } }
I20260928 12:30:53.594990 10708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69eb8157527f4485a5cda8a996f6804b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb8157527f4485a5cda8a996f6804b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33151 } } }
I20260928 12:30:53.595060 10708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:53.595194 10708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69eb8157527f4485a5cda8a996f6804b. Latest consensus state: current_term: 1 leader_uuid: "69eb8157527f4485a5cda8a996f6804b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb8157527f4485a5cda8a996f6804b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33151 } } }
I20260928 12:30:53.595248 10708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b [sys.catalog]: This master's current role is: LEADER
W20260928 12:30:53.596426 10724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:30:53.596481 10724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:30:53.596531 10714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:53.596673 10714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:53.597739 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:53.587372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.8.13.62:33151)
0928 12:30:53.587571 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:53.587582 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:53.587618 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:30:53.588158 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:53.588167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:53.588240 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:53.588399 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:53.589004 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:53.590356 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:53.591171 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:53.591796 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:53.591922 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:53.594208 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:53.594213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:53.594213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:53.594227 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:53.596236 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:53.596598 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:53.596602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:53.596602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:53.596695 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:53.596900 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:53.596901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:53.596901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:53.597139 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:53.597286 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:53.597392 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 12:30:53.597527 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260928 12:30:53.598189 10714 catalog_manager.cc:1347] Generated new cluster ID: d37991ac9f064385b8b6bfc22983984c
I20260928 12:30:53.598232 10714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:53.606225 10714 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:53.606725 10714 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:53.615773 10714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69eb8157527f4485a5cda8a996f6804b: Generated new TSK 0
I20260928 12:30:53.615998 10714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:30:53.653098 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33151
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:53.738613 10729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:53.738833 10729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:53.738857 10729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:53.740144 10729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:53.740206 10729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:53.740236 10729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:30:53.741920 10729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:53.742190 10729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:53.742439 10729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:53.747865 10735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:53.747865 10737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:53.747907 10734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:53.747923 10729 server_base.cc:1034] running on GCE node
I20260928 12:30:53.748283 10729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:53.748560 10729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:53.749707 10729 hybrid_clock.cc:648] HybridClock initialized: now 1790598653749694 us; error 31 us; skew 500 ppm
Sep 28 12:30:53 dist-test-slave-g31s krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598653, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:53.752362 10729 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:30:53.752826 10729 webserver.cc:460] Webserver started at http://127.8.13.1:35073/ using document root <none> and password file <none>
I20260928 12:30:53.753062 10729 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:53.753113 10729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:53.753240 10729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:53.754446 10729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "f92e098832ab494daec87e6c50374cf7"
format_stamp: "Formatted at 2026-09-28 12:30:53 on dist-test-slave-g31s"
server_key: "b146cfa67cdd4fd2baf55f4f9dcf4a4c"
server_key_iv: "10fc54132cd0ade80aa47a4cd5fe383e"
server_key_version: "encryptionkey@0"
I20260928 12:30:53.754793 10729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "f92e098832ab494daec87e6c50374cf7"
format_stamp: "Formatted at 2026-09-28 12:30:53 on dist-test-slave-g31s"
server_key: "b146cfa67cdd4fd2baf55f4f9dcf4a4c"
server_key_iv: "10fc54132cd0ade80aa47a4cd5fe383e"
server_key_version: "encryptionkey@0"
I20260928 12:30:53.756110 10729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:53.756901 10744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:53.757135 10729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:53.757238 10729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "f92e098832ab494daec87e6c50374cf7"
format_stamp: "Formatted at 2026-09-28 12:30:53 on dist-test-slave-g31s"
server_key: "b146cfa67cdd4fd2baf55f4f9dcf4a4c"
server_key_iv: "10fc54132cd0ade80aa47a4cd5fe383e"
server_key_version: "encryptionkey@0"
I20260928 12:30:53.757308 10729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:53.770012 10729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:53.772377 10729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:53.772529 10729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:53.772775 10729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:53.773116 10729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:53.773155 10729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:53.773187 10729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:53.773203 10729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:53.778904 10729 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:41281
I20260928 12:30:53.778990 10857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:41281 every 8 connection(s)
I20260928 12:30:53.779565 10729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:30:53 dist-test-slave-g31s krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598653, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:53.788582 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 10729
I20260928 12:30:53.788678 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:30:53.788849 8244 external_mini_cluster.cc:1428] Setting key 9b6ce58c56f765f890df7565b7e56066
I20260928 12:30:53.789214 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:53.780366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:53855 (local address 127.8.13.62:33151)
0928 12:30:53.780447 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:53.780449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:53.780646 (+ 197us) server_negotiation.cc:407] Connection header received
0928 12:30:53.781134 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:53.781135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:53.781149 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:53.781213 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:53.782260 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:53.782809 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:53.783833 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:53.784043 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:53.784106 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:53.786300 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:53.786302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:53.786302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:53.786310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:53.787676 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:53.788141 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:53.788141 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:53.788142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:53.788190 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:53.788470 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:53.788471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:53.788471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:53.788587 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:53.788633 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:53.788999 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 12:30:53.789113 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:30:53.789414 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:53.780334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33151 (local address 127.8.13.1:53855)
0928 12:30:53.780493 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:53.780544 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:53.781024 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:53.781270 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:53.781273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:53.781925 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:53.782144 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:53.782149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:53.782891 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:53.782891 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:53.783738 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:53.783740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:53.784117 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:53.784117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:53.784226 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:53.784733 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:53.784742 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:53.786182 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:53.787784 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:53.787795 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:53.787797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:53.788066 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:53.788268 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:53.788269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:53.788269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:53.788396 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:53.788640 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:53.788647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:53.788838 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 12:30:53.788955 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 12:30:53.789133 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260928 12:30:53.789960 10858 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33151
I20260928 12:30:53.790072 10858 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:53.790266 10858 heartbeater.cc:511] Master 127.8.13.62:33151 requested a full tablet report, sending...
I20260928 12:30:53.790829 10667 ts_manager.cc:194] Registered new tserver with Master: f92e098832ab494daec87e6c50374cf7 (127.8.13.1:41281)
I20260928 12:30:53.791471 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:53855
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:30:53.814901 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33151
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:53.900027 10863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:53.900242 10863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:53.900269 10863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:53.901546 10863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:53.901600 10863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:53.901629 10863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:30:53.903257 10863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:53.903513 10863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:53.903743 10863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:53.909019 10871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:53.909090 10868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:53.909024 10869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:53.909096 10863 server_base.cc:1034] running on GCE node
I20260928 12:30:53.909535 10863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:53.909773 10863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:53.910960 10863 hybrid_clock.cc:648] HybridClock initialized: now 1790598653910943 us; error 30 us; skew 500 ppm
Sep 28 12:30:53 dist-test-slave-g31s krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598653, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:53.913343 10863 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:30:53.913790 10863 webserver.cc:460] Webserver started at http://127.8.13.2:37753/ using document root <none> and password file <none>
I20260928 12:30:53.914019 10863 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:53.914058 10863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:53.914151 10863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:53.915143 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "ac8964b1f09a467398418c50377f43e4"
format_stamp: "Formatted at 2026-09-28 12:30:53 on dist-test-slave-g31s"
server_key: "466b947694e1c77e23cd8310be2cfc63"
server_key_iv: "808ae843b6ece231427582ccbb238428"
server_key_version: "encryptionkey@0"
I20260928 12:30:53.915436 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "ac8964b1f09a467398418c50377f43e4"
format_stamp: "Formatted at 2026-09-28 12:30:53 on dist-test-slave-g31s"
server_key: "466b947694e1c77e23cd8310be2cfc63"
server_key_iv: "808ae843b6ece231427582ccbb238428"
server_key_version: "encryptionkey@0"
I20260928 12:30:53.916682 10863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:53.917438 10878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:53.917637 10863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:53.917820 10863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "ac8964b1f09a467398418c50377f43e4"
format_stamp: "Formatted at 2026-09-28 12:30:53 on dist-test-slave-g31s"
server_key: "466b947694e1c77e23cd8310be2cfc63"
server_key_iv: "808ae843b6ece231427582ccbb238428"
server_key_version: "encryptionkey@0"
I20260928 12:30:53.917904 10863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:53.938417 10863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:53.941967 10863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:53.942128 10863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:53.942365 10863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:53.942683 10863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:53.942716 10863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:53.942749 10863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:53.942772 10863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:53.949209 10863 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:40823
I20260928 12:30:53.949290 10991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:40823 every 8 connection(s)
I20260928 12:30:53.949615 10863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:30:53.949885 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 10863
I20260928 12:30:53.949982 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:30:53.950168 8244 external_mini_cluster.cc:1428] Setting key 6c41be5cbecbed5409e7a93a9406d649
Sep 28 12:30:53 dist-test-slave-g31s krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598653, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
I20260928 12:30:53.961951 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:53.950487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:59505 (local address 127.8.13.62:33151)
0928 12:30:53.950542 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:53.950543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:53.950794 (+ 251us) server_negotiation.cc:407] Connection header received
0928 12:30:53.951444 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:53.951446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:53.951456 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:53.951531 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:53.952779 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:53.953311 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:53.954966 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:53.955189 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:53.955255 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:53.958111 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:53.958115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:53.958115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:53.958127 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:53.959944 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:53.960615 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:53.960617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:53.960618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:53.960747 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:53.961131 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:53.961133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:53.961133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:53.961294 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:53.961367 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:53.961706 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 12:30:53.961843 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
I20260928 12:30:53.962212 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:53.950485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33151 (local address 127.8.13.2:59505)
0928 12:30:53.950647 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:53.950698 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:53.951311 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:53.951608 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:53.951613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:53.952401 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:53.952667 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:53.952673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:53.953406 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:53.953407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:53.954793 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:53.954806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:53.955282 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:53.955284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:53.955451 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:53.956062 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:53.956087 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:53.957918 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:53.960117 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:53.960133 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:53.960137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:53.960507 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:53.960856 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:53.960857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:53.960858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:53.961030 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:53.961363 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:53.961372 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:53.961559 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 12:30:53.961668 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 12:30:53.961948 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260928 12:30:53.962710 10992 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33151
I20260928 12:30:53.962817 10992 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:53.963029 10992 heartbeater.cc:511] Master 127.8.13.62:33151 requested a full tablet report, sending...
I20260928 12:30:53.963456 10667 ts_manager.cc:194] Registered new tserver with Master: ac8964b1f09a467398418c50377f43e4 (127.8.13.2:40823)
I20260928 12:30:53.963907 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:59505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:30:53.978587 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33151
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:54.059878 10997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:54.060096 10997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:54.060129 10997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:54.061429 10997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:54.061481 10997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:54.061522 10997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:30:54.063131 10997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:54.063408 10997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:54.063633 10997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:54.068712 11005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:54.068756 11003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:54.068717 11002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:54.069012 10997 server_base.cc:1034] running on GCE node
I20260928 12:30:54.069278 10997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:54.069530 10997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:54.070675 10997 hybrid_clock.cc:648] HybridClock initialized: now 1790598654070660 us; error 29 us; skew 500 ppm
Sep 28 12:30:54 dist-test-slave-g31s krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598654, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:54.073248 10997 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:30:54.073702 10997 webserver.cc:460] Webserver started at http://127.8.13.3:43999/ using document root <none> and password file <none>
I20260928 12:30:54.073959 10997 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:54.074020 10997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:54.074146 10997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:54.075207 10997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "f9767c5cc1ff4e2ab8dbd7bce2da597b"
format_stamp: "Formatted at 2026-09-28 12:30:54 on dist-test-slave-g31s"
server_key: "54f495d38360954c26a3f3eef3de2341"
server_key_iv: "8b28948c26516b329c76d848b459c936"
server_key_version: "encryptionkey@0"
I20260928 12:30:54.075544 10997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "f9767c5cc1ff4e2ab8dbd7bce2da597b"
format_stamp: "Formatted at 2026-09-28 12:30:54 on dist-test-slave-g31s"
server_key: "54f495d38360954c26a3f3eef3de2341"
server_key_iv: "8b28948c26516b329c76d848b459c936"
server_key_version: "encryptionkey@0"
I20260928 12:30:54.076813 10997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:54.077553 11012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:54.077697 10997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:54.077785 10997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "f9767c5cc1ff4e2ab8dbd7bce2da597b"
format_stamp: "Formatted at 2026-09-28 12:30:54 on dist-test-slave-g31s"
server_key: "54f495d38360954c26a3f3eef3de2341"
server_key_iv: "8b28948c26516b329c76d848b459c936"
server_key_version: "encryptionkey@0"
I20260928 12:30:54.077884 10997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:54.088455 10997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:54.090782 10997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:54.090925 10997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:54.091146 10997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:54.091439 10997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:54.091475 10997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:54.091504 10997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:54.091529 10997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:54.098115 10997 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:34541
I20260928 12:30:54.098214 11125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:34541 every 8 connection(s)
I20260928 12:30:54.098532 10997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:30:54.103998 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 10997
I20260928 12:30:54.104130 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:30:54.104315 8244 external_mini_cluster.cc:1428] Setting key 7edebff9a94abf660c89d9c4d9f4096b
Sep 28 12:30:54 dist-test-slave-g31s krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598654, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:54.109581 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:54.099311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:57931 (local address 127.8.13.62:33151)
0928 12:30:54.099435 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:54.099438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:54.099618 (+ 180us) server_negotiation.cc:407] Connection header received
0928 12:30:54.100055 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:54.100056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:54.100069 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:54.100132 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:54.101087 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:54.101614 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:54.103103 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:54.103319 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:54.103416 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:54.106617 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:54.106618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:54.106618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:54.106627 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:54.108000 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:54.108501 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:54.108502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:54.108503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:54.108601 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:54.108884 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:54.108886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:54.108887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:54.109043 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:54.109115 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:54.109405 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 12:30:54.109490 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 12:30:54.109885 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:54.099255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33151 (local address 127.8.13.3:57931)
0928 12:30:54.099428 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:54.099478 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:54.099962 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:54.100187 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:54.100190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:54.100772 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:54.100984 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:54.100990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:54.101712 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:54.101713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:54.102986 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:54.102989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:54.103461 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:54.103463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:54.103643 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:54.104418 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:54.104430 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:54.106493 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:54.108102 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:54.108112 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:54.108114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:54.108386 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:54.108681 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:54.108682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:54.108682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:54.108790 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:54.109135 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:54.109141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:54.109289 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 12:30:54.109388 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 12:30:54.109546 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260928 12:30:54.110333 11126 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33151
I20260928 12:30:54.110435 11126 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:54.110623 11126 heartbeater.cc:511] Master 127.8.13.62:33151 requested a full tablet report, sending...
I20260928 12:30:54.110963 10667 ts_manager.cc:194] Registered new tserver with Master: f9767c5cc1ff4e2ab8dbd7bce2da597b (127.8.13.3:34541)
I20260928 12:30:54.111336 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:57931
I20260928 12:30:54.116495 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:30:54.127326 10706 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:54.126437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.8.13.62:33151)
0928 12:30:54.126518 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:54.126520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:54.126595 (+ 75us) server_negotiation.cc:407] Connection header received
0928 12:30:54.126662 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:54.126664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:54.126674 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:54.126721 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:54.127250 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52562: BlockingRecv error: recv got EOF from 127.0.0.1:52562 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 12:30:54.128167 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 10729
I20260928 12:30:54.132367 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 10863
I20260928 12:30:54.136190 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 10997
I20260928 12:30:54.140314 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 10635
2026-09-28T12:30:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1757 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:54 dist-test-slave-g31s krb5kdc[11138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:54 dist-test-slave-g31s krb5kdc[11138](info): set up 2 sockets
Sep 28 12:30:54 dist-test-slave-g31s krb5kdc[11138](info): commencing operation
krb5kdc: starting...
W20260928 12:30:55.102198 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:30:55 dist-test-slave-g31s krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:30:55.218812 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:36249
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:36249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:55.302346 11151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:55.302591 11151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:55.302615 11151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:55.303918 11151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:55.303972 11151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:55.303987 11151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:55.303998 11151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:55.304009 11151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:55.306334 11151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:36249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:36249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:55.306638 11151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:55.306869 11151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:55.311852 11159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:55.311856 11156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:55.311856 11157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:55.311918 11151 server_base.cc:1034] running on GCE node
I20260928 12:30:55.312270 11151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:55.312559 11151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:55.313714 11151 hybrid_clock.cc:648] HybridClock initialized: now 1790598655313696 us; error 35 us; skew 500 ppm
Sep 28 12:30:55 dist-test-slave-g31s krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598655, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:55.316103 11151 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:30:55.316501 11151 webserver.cc:460] Webserver started at http://127.8.13.62:39649/ using document root <none> and password file <none>
I20260928 12:30:55.316680 11151 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:55.316720 11151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:55.316802 11151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:55.317847 11151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "3a93468776ee4bf09ed6573e5a893a62"
format_stamp: "Formatted at 2026-09-28 12:30:55 on dist-test-slave-g31s"
server_key: "8aef293a7f91983909f4dca87e322e06"
server_key_iv: "b85dc64eb54937c70f5fbc9dfcd38bc6"
server_key_version: "encryptionkey@0"
I20260928 12:30:55.318161 11151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "3a93468776ee4bf09ed6573e5a893a62"
format_stamp: "Formatted at 2026-09-28 12:30:55 on dist-test-slave-g31s"
server_key: "8aef293a7f91983909f4dca87e322e06"
server_key_iv: "b85dc64eb54937c70f5fbc9dfcd38bc6"
server_key_version: "encryptionkey@0"
I20260928 12:30:55.319320 11151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:55.319952 11166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:55.320099 11151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:55.320173 11151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "3a93468776ee4bf09ed6573e5a893a62"
format_stamp: "Formatted at 2026-09-28 12:30:55 on dist-test-slave-g31s"
server_key: "8aef293a7f91983909f4dca87e322e06"
server_key_iv: "b85dc64eb54937c70f5fbc9dfcd38bc6"
server_key_version: "encryptionkey@0"
I20260928 12:30:55.320236 11151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:55.339821 11151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:55.342337 11151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:55.342466 11151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:55.346349 11151 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:36249
I20260928 12:30:55.346431 11218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:36249 every 8 connection(s)
I20260928 12:30:55.346776 11151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:55.348083 11219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:55.350710 11219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62: Bootstrap starting.
I20260928 12:30:55.351330 11219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:55.351619 11219 log.cc:824] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:55.352355 11219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62: No bootstrap required, opened a new log
I20260928 12:30:55.353834 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 11151
I20260928 12:30:55.353940 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:30:55.354116 8244 external_mini_cluster.cc:1428] Setting key a0c5031055bbb21323def6825418042c
I20260928 12:30:55.354440 11219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93468776ee4bf09ed6573e5a893a62" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 36249 } }
I20260928 12:30:55.354678 11219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:55.354735 11219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a93468776ee4bf09ed6573e5a893a62, State: Initialized, Role: FOLLOWER
I20260928 12:30:55.354894 11219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93468776ee4bf09ed6573e5a893a62" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 36249 } }
I20260928 12:30:55.354974 11219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:55.355000 11219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:55.355041 11219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:55.355698 11219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93468776ee4bf09ed6573e5a893a62" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 36249 } }
I20260928 12:30:55.355821 11219 leader_election.cc:302] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a93468776ee4bf09ed6573e5a893a62; no voters:
I20260928 12:30:55.356024 11219 leader_election.cc:288] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:55.356055 11224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:55.356221 11224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [term 1 LEADER]: Becoming Leader. State: Replica: 3a93468776ee4bf09ed6573e5a893a62, State: Running, Role: LEADER
I20260928 12:30:55.356323 11219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:55.356355 11224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93468776ee4bf09ed6573e5a893a62" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 36249 } }
I20260928 12:30:55.356870 11225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a93468776ee4bf09ed6573e5a893a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93468776ee4bf09ed6573e5a893a62" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 36249 } } }
I20260928 12:30:55.356950 11225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:55.357193 11226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a93468776ee4bf09ed6573e5a893a62. Latest consensus state: current_term: 1 leader_uuid: "3a93468776ee4bf09ed6573e5a893a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93468776ee4bf09ed6573e5a893a62" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 36249 } } }
I20260928 12:30:55.357538 11226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:55.357576 11233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:55.358163 11233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:55.360157 11233 catalog_manager.cc:1347] Generated new cluster ID: 776a224b00414dffaf652025d6d8ca7f
I20260928 12:30:55.360212 11233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:30:55 dist-test-slave-g31s krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:55.366634 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:55.354710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.8.13.62:36249)
0928 12:30:55.355322 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:55.355331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:55.355368 (+ 37us) server_negotiation.cc:407] Connection header received
0928 12:30:55.355491 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:55.355497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:55.355545 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:55.355672 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:55.356608 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:55.358094 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:55.359149 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:55.359659 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:55.359766 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:55.362746 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:55.362751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:55.362752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:55.362766 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:55.364852 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:55.365319 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:55.365322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:55.365322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:55.365443 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:55.365654 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:55.365654 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:55.365655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:55.366002 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:55.366169 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:55.366263 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 12:30:55.366413 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":508,"threads_started":1}
I20260928 12:30:55.367035 11233 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:55.367476 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:55.367734 11233 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:55.383850 11233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a93468776ee4bf09ed6573e5a893a62: Generated new TSK 0
I20260928 12:30:55.384136 11233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:55.388222 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:55.383812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.8.13.62:36249)
0928 12:30:55.383871 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:55.383874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:55.384040 (+ 166us) server_negotiation.cc:407] Connection header received
0928 12:30:55.384166 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:55.384167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:55.384184 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:55.384246 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:55.384845 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:55.385394 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:55.386587 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:55.386831 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:55.386912 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:55.387735 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:55.387737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:55.387737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:55.387977 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:55.387996 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:55.388002 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:55.388048 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:55.388127 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 12:30:55.388154 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:30:55.390811 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 11151
2026-09-28T12:30:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1248 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:55 dist-test-slave-g31s krb5kdc[11252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:55 dist-test-slave-g31s krb5kdc[11252](info): set up 2 sockets
Sep 28 12:30:55 dist-test-slave-g31s krb5kdc[11252](info): commencing operation
krb5kdc: starting...
W20260928 12:30:56.349727 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:30:56 dist-test-slave-g31s krb5kdc[11252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:30:56.469185 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:39163
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:39163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:56.550179 11265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:56.550451 11265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:56.550486 11265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:56.551923 11265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:56.551996 11265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:56.552016 11265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:56.552033 11265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:56.552048 11265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:56.553887 11265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:39163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:39163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:56.554204 11265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:56.554445 11265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:56.559737 11271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:56.559810 11265 server_base.cc:1034] running on GCE node
W20260928 12:30:56.559782 11273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:56.559782 11270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:56.560143 11265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:56.560451 11265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:56.561632 11265 hybrid_clock.cc:648] HybridClock initialized: now 1790598656561610 us; error 44 us; skew 500 ppm
Sep 28 12:30:56 dist-test-slave-g31s krb5kdc[11252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598656, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:56.564585 11265 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:30:56.565078 11265 webserver.cc:460] Webserver started at http://127.8.13.62:43451/ using document root <none> and password file <none>
I20260928 12:30:56.565291 11265 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:56.565339 11265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:56.565481 11265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:56.566675 11265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "47f93558040d4830b737b67a05800516"
format_stamp: "Formatted at 2026-09-28 12:30:56 on dist-test-slave-g31s"
server_key: "fd514827822d5064aff2c82d95ad2c3a"
server_key_iv: "5ba907db02b28998ebbdd7de1056c23c"
server_key_version: "encryptionkey@0"
I20260928 12:30:56.567036 11265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "47f93558040d4830b737b67a05800516"
format_stamp: "Formatted at 2026-09-28 12:30:56 on dist-test-slave-g31s"
server_key: "fd514827822d5064aff2c82d95ad2c3a"
server_key_iv: "5ba907db02b28998ebbdd7de1056c23c"
server_key_version: "encryptionkey@0"
I20260928 12:30:56.568477 11265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:56.569295 11280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:56.569509 11265 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:56.569602 11265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "47f93558040d4830b737b67a05800516"
format_stamp: "Formatted at 2026-09-28 12:30:56 on dist-test-slave-g31s"
server_key: "fd514827822d5064aff2c82d95ad2c3a"
server_key_iv: "5ba907db02b28998ebbdd7de1056c23c"
server_key_version: "encryptionkey@0"
I20260928 12:30:56.569665 11265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:56.578828 11265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:56.581406 11265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:56.581534 11265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:56.585937 11265 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:39163
I20260928 12:30:56.585971 11332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:39163 every 8 connection(s)
I20260928 12:30:56.586334 11265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:56.587348 11333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:56.590634 11333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516: Bootstrap starting.
I20260928 12:30:56.591437 11333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:56.591744 11333 log.cc:824] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:56.592839 11333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516: No bootstrap required, opened a new log
I20260928 12:30:56.594524 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 11265
I20260928 12:30:56.594638 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:30:56.594808 8244 external_mini_cluster.cc:1428] Setting key d77b620da8077a4e85d8e207bf870610
I20260928 12:30:56.595058 11333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f93558040d4830b737b67a05800516" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39163 } }
I20260928 12:30:56.595252 11333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:56.595286 11333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f93558040d4830b737b67a05800516, State: Initialized, Role: FOLLOWER
I20260928 12:30:56.595448 11333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f93558040d4830b737b67a05800516" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39163 } }
I20260928 12:30:56.595541 11333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:56.595598 11333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:56.595639 11333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:56.596495 11333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f93558040d4830b737b67a05800516" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39163 } }
I20260928 12:30:56.596626 11333 leader_election.cc:302] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47f93558040d4830b737b67a05800516; no voters:
I20260928 12:30:56.596822 11333 leader_election.cc:288] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:56.596856 11338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:56.596953 11338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [term 1 LEADER]: Becoming Leader. State: Replica: 47f93558040d4830b737b67a05800516, State: Running, Role: LEADER
I20260928 12:30:56.597112 11333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:56.597090 11338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f93558040d4830b737b67a05800516" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39163 } }
I20260928 12:30:56.597464 11340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47f93558040d4830b737b67a05800516. Latest consensus state: current_term: 1 leader_uuid: "47f93558040d4830b737b67a05800516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f93558040d4830b737b67a05800516" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39163 } } }
I20260928 12:30:56.597664 11340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:56.597610 11339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47f93558040d4830b737b67a05800516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f93558040d4830b737b67a05800516" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39163 } } }
I20260928 12:30:56.597923 11339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:56.598057 11343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:56.598609 11343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:56.600379 11343 catalog_manager.cc:1347] Generated new cluster ID: 6a95a22a818143239766cbfcefe58dd5
I20260928 12:30:56.600436 11343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:30:56 dist-test-slave-g31s krb5kdc[11252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:56.605269 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:56.595392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60912 (local address 127.8.13.62:39163)
0928 12:30:56.595653 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:56.595664 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:56.595704 (+ 40us) server_negotiation.cc:407] Connection header received
0928 12:30:56.595808 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:56.595814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:56.595858 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:56.595966 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:56.596752 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:56.598082 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:56.599200 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:56.599717 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:56.599820 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:56.602227 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:56.602233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:56.602233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:56.602248 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:56.603838 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:56.604169 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:56.604172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:56.604173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:56.604271 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:56.604446 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:56.604447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:56.604447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:56.604693 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:56.604833 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:56.604934 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 12:30:56.605074 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
I20260928 12:30:56.614029 11343 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:56.614677 11343 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:56.616899 11343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47f93558040d4830b737b67a05800516: Generated new TSK 0
I20260928 12:30:56.617074 11343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:30:56.649859 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:39163
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:56.734510 11359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:56.734864 11359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:56.734915 11359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:56.736352 11359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:56.736415 11359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:56.736446 11359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:30:56.738139 11359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:39163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:56.738389 11359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:56.738631 11359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:56.743849 11365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:56.743862 11367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:56.743865 11364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:56.744210 11359 server_base.cc:1034] running on GCE node
I20260928 12:30:56.744436 11359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:56.744673 11359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:56.745875 11359 hybrid_clock.cc:648] HybridClock initialized: now 1790598656745854 us; error 29 us; skew 500 ppm
Sep 28 12:30:56 dist-test-slave-g31s krb5kdc[11252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598656, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:56.748535 11359 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:30:56.748978 11359 webserver.cc:460] Webserver started at http://127.8.13.1:37583/ using document root <none> and password file <none>
I20260928 12:30:56.749162 11359 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:56.749202 11359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:56.749300 11359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:56.750495 11359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "2705d5b426e8425392495312799d3989"
format_stamp: "Formatted at 2026-09-28 12:30:56 on dist-test-slave-g31s"
server_key: "4f99253d0212ace1783723f4f5cc7181"
server_key_iv: "a305b033c11d22ff56b4e2f672c2f1ef"
server_key_version: "encryptionkey@0"
I20260928 12:30:56.750833 11359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "2705d5b426e8425392495312799d3989"
format_stamp: "Formatted at 2026-09-28 12:30:56 on dist-test-slave-g31s"
server_key: "4f99253d0212ace1783723f4f5cc7181"
server_key_iv: "a305b033c11d22ff56b4e2f672c2f1ef"
server_key_version: "encryptionkey@0"
I20260928 12:30:56.752214 11359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:56.753041 11374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:56.753222 11359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:56.753317 11359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "2705d5b426e8425392495312799d3989"
format_stamp: "Formatted at 2026-09-28 12:30:56 on dist-test-slave-g31s"
server_key: "4f99253d0212ace1783723f4f5cc7181"
server_key_iv: "a305b033c11d22ff56b4e2f672c2f1ef"
server_key_version: "encryptionkey@0"
I20260928 12:30:56.753391 11359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:56.770298 11359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:56.772619 11359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:56.772748 11359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:56.772961 11359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:56.773291 11359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:56.773326 11359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:56.773350 11359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:56.773373 11359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:56.779695 11359 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:35411
I20260928 12:30:56.779788 11487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:35411 every 8 connection(s)
I20260928 12:30:56.780143 11359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:30:56.785226 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 11359
I20260928 12:30:56.785317 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:30:56.785509 8244 external_mini_cluster.cc:1428] Setting key 65b30f17283886cb521d09dedfe65bab
Sep 28 12:30:56 dist-test-slave-g31s krb5kdc[11252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598656, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:56.791507 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:56.780988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:40023 (local address 127.8.13.62:39163)
0928 12:30:56.781061 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:56.781063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:56.781273 (+ 210us) server_negotiation.cc:407] Connection header received
0928 12:30:56.781766 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:56.781767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:56.781782 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:56.781897 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:56.782958 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:56.783518 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:56.785246 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:56.785546 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:56.785672 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:56.788571 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:56.788574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:56.788574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:56.788584 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:56.790041 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:56.790542 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:56.790544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:56.790544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:56.790600 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:56.790836 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:56.790836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:56.790837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:56.790957 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:56.791017 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:56.791311 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 12:30:56.791409 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:30:56.791712 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:56.780911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:39163 (local address 127.8.13.1:40023)
0928 12:30:56.781131 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:56.781175 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:56.781665 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:56.781984 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:56.781988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:56.782601 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:56.782835 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:56.782841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:56.783746 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:56.783749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:56.785037 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:56.785045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:56.785741 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:56.785748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:56.786052 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:56.786761 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:56.786773 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:56.788432 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:56.790149 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:56.790163 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:56.790167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:56.790464 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:56.790660 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:56.790661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:56.790662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:56.790777 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:56.791015 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:56.791021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:56.791184 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 12:30:56.791280 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 12:30:56.791468 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260928 12:30:56.792258 11488 heartbeater.cc:347] Connected to a master server at 127.8.13.62:39163
I20260928 12:30:56.792358 11488 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:56.792551 11488 heartbeater.cc:511] Master 127.8.13.62:39163 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
I20260928 12:30:56.793462 11297 ts_manager.cc:194] Registered new tserver with Master: 2705d5b426e8425392495312799d3989 (127.8.13.1:35411)
I20260928 12:30:56.794159 11297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:40023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:30:56.812390 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:39163
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:56.896198 11493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:56.896450 11493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:56.896487 11493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:56.897977 11493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:56.898041 11493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:56.898083 11493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:30:56.899840 11493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:39163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:56.900174 11493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:56.900413 11493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:56.905666 11499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:56.905822 11493 server_base.cc:1034] running on GCE node
W20260928 12:30:56.905676 11498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:56.905928 11501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:56.906123 11493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:56.906392 11493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:56.907655 11493 hybrid_clock.cc:648] HybridClock initialized: now 1790598656907592 us; error 29 us; skew 500 ppm
Sep 28 12:30:56 dist-test-slave-g31s krb5kdc[11252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598656, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:56.914027 11493 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:30:56.914551 11493 webserver.cc:460] Webserver started at http://127.8.13.2:45955/ using document root <none> and password file <none>
I20260928 12:30:56.914758 11493 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:56.914806 11493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:56.914932 11493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:56.916041 11493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "816659acf0474b16a0a2a16527affe90"
format_stamp: "Formatted at 2026-09-28 12:30:56 on dist-test-slave-g31s"
server_key: "4a8d9e6c690acaf93b50f86f03d62a8e"
server_key_iv: "c1631094bd8657096a6d302ea1c504b8"
server_key_version: "encryptionkey@0"
I20260928 12:30:56.916383 11493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "816659acf0474b16a0a2a16527affe90"
format_stamp: "Formatted at 2026-09-28 12:30:56 on dist-test-slave-g31s"
server_key: "4a8d9e6c690acaf93b50f86f03d62a8e"
server_key_iv: "c1631094bd8657096a6d302ea1c504b8"
server_key_version: "encryptionkey@0"
I20260928 12:30:56.917728 11493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:30:56.918697 11508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:56.919097 11493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:56.919253 11493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "816659acf0474b16a0a2a16527affe90"
format_stamp: "Formatted at 2026-09-28 12:30:56 on dist-test-slave-g31s"
server_key: "4a8d9e6c690acaf93b50f86f03d62a8e"
server_key_iv: "c1631094bd8657096a6d302ea1c504b8"
server_key_version: "encryptionkey@0"
I20260928 12:30:56.919356 11493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:56.930807 11493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:56.933203 11493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:56.933396 11493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:56.933699 11493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:56.934139 11493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:56.934176 11493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:56.934211 11493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:56.934226 11493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:56.941866 11493 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:44205
I20260928 12:30:56.941929 11621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:44205 every 8 connection(s)
I20260928 12:30:56.942480 11493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:30:56.948563 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 11493
I20260928 12:30:56.948711 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:30:56.948935 8244 external_mini_cluster.cc:1428] Setting key 60a7b4464320e0d3117ad24529fc00a4
Sep 28 12:30:56 dist-test-slave-g31s krb5kdc[11252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598656, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:56.954869 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:56.943378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:52189 (local address 127.8.13.62:39163)
0928 12:30:56.943516 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:56.943519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:56.943861 (+ 342us) server_negotiation.cc:407] Connection header received
0928 12:30:56.944436 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:56.944438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:56.944452 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:56.944513 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:56.945597 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:56.946312 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:56.947866 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:56.948094 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:56.948160 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:56.951371 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:56.951374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:56.951375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:56.951386 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:56.952756 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:56.953404 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:56.953408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:56.953408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:56.953537 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:56.953950 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:56.953952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:56.953952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:56.954117 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:56.954206 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:56.954680 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 12:30:56.954789 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 12:30:56.955081 11624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:56.943303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:39163 (local address 127.8.13.2:52189)
0928 12:30:56.943592 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:56.943660 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:56.944328 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:56.944593 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:56.944598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:56.945251 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:56.945488 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:56.945494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:56.946487 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:56.946489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:56.947743 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:56.947748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:56.948174 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:56.948175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:56.948358 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:56.949206 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:56.949224 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:56.951219 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:56.952865 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:56.952876 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:56.952879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:56.953239 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:56.953660 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:56.953661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:56.953661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:56.953869 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:56.954209 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:56.954216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:56.954434 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 12:30:56.954581 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 12:30:56.954800 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":141,"threads_started":1}
I20260928 12:30:56.955734 11622 heartbeater.cc:347] Connected to a master server at 127.8.13.62:39163
I20260928 12:30:56.955858 11622 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:56.956091 11622 heartbeater.cc:511] Master 127.8.13.62:39163 requested a full tablet report, sending...
I20260928 12:30:56.956678 11297 ts_manager.cc:194] Registered new tserver with Master: 816659acf0474b16a0a2a16527affe90 (127.8.13.2:44205)
I20260928 12:30:56.957162 11297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:52189
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:30:56.977836 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:39163
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:57.063118 11627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:57.063365 11627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:57.063421 11627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:57.064673 11627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:57.064718 11627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:57.064746 11627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:30:57.066288 11627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:39163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:57.066529 11627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:57.066730 11627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:57.071826 11632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:57.071830 11635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:57.071899 11627 server_base.cc:1034] running on GCE node
W20260928 12:30:57.071830 11633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:57.072232 11627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:57.072535 11627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:57.073716 11627 hybrid_clock.cc:648] HybridClock initialized: now 1790598657073701 us; error 27 us; skew 500 ppm
Sep 28 12:30:57 dist-test-slave-g31s krb5kdc[11252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598657, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:57.076622 11627 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:30:57.077107 11627 webserver.cc:460] Webserver started at http://127.8.13.3:38583/ using document root <none> and password file <none>
I20260928 12:30:57.077308 11627 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:57.077351 11627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:57.077447 11627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:57.078621 11627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "bd929442052b4ca2b946cbb3ce3110c7"
format_stamp: "Formatted at 2026-09-28 12:30:57 on dist-test-slave-g31s"
server_key: "ae7c2b7776391ef7d367d4c2672d2eb6"
server_key_iv: "4e6585fda1d71c00c65603b29dbbdd1b"
server_key_version: "encryptionkey@0"
I20260928 12:30:57.078964 11627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "bd929442052b4ca2b946cbb3ce3110c7"
format_stamp: "Formatted at 2026-09-28 12:30:57 on dist-test-slave-g31s"
server_key: "ae7c2b7776391ef7d367d4c2672d2eb6"
server_key_iv: "4e6585fda1d71c00c65603b29dbbdd1b"
server_key_version: "encryptionkey@0"
I20260928 12:30:57.080386 11627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:57.081386 11642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:57.081604 11627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:57.081694 11627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "bd929442052b4ca2b946cbb3ce3110c7"
format_stamp: "Formatted at 2026-09-28 12:30:57 on dist-test-slave-g31s"
server_key: "ae7c2b7776391ef7d367d4c2672d2eb6"
server_key_iv: "4e6585fda1d71c00c65603b29dbbdd1b"
server_key_version: "encryptionkey@0"
I20260928 12:30:57.081764 11627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:57.103111 11627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:57.106042 11627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:57.106182 11627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:57.106382 11627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:30:57.106668 11627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:30:57.106703 11627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:57.106726 11627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:30:57.106742 11627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:57.112514 11627 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:39029
I20260928 12:30:57.112591 11755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:39029 every 8 connection(s)
I20260928 12:30:57.112957 11627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:30:57.113122 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 11627
I20260928 12:30:57.113216 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:30:57.113433 8244 external_mini_cluster.cc:1428] Setting key 8456015d5c1334ddf94dfee84d07049c
Sep 28 12:30:57 dist-test-slave-g31s krb5kdc[11252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598657, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:57.122247 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:57.113888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:52251 (local address 127.8.13.62:39163)
0928 12:30:57.113931 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:57.113933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:57.114211 (+ 278us) server_negotiation.cc:407] Connection header received
0928 12:30:57.114731 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:57.114732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:57.114742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:57.114791 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:57.115738 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:57.116296 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:57.117486 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:57.117671 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:57.117722 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:57.119806 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:57.119808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:57.119808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:57.119815 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:57.120882 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:57.121227 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:57.121228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:57.121229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:57.121281 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:57.121568 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:57.121569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:57.121570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:57.121691 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:57.121759 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:57.122101 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 12:30:57.122176 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:30:57.122366 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:57.113899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:39163 (local address 127.8.13.3:52251)
0928 12:30:57.114071 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:57.114121 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:57.114644 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:57.114865 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:57.114877 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:57.115442 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:57.115638 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:57.115642 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:57.116395 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:57.116396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:57.117388 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:57.117391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:57.117729 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:57.117729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:57.117856 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:57.118359 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:57.118367 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:57.119725 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:57.120949 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:57.120954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:57.120956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:57.121159 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:57.121363 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:57.121364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:57.121364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:57.121478 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:57.121771 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:57.121777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:57.121955 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 12:30:57.122049 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 12:30:57.122180 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260928 12:30:57.122920 11756 heartbeater.cc:347] Connected to a master server at 127.8.13.62:39163
I20260928 12:30:57.123039 11756 heartbeater.cc:464] Registering TS with master...
I20260928 12:30:57.123201 11756 heartbeater.cc:511] Master 127.8.13.62:39163 requested a full tablet report, sending...
I20260928 12:30:57.123553 11297 ts_manager.cc:194] Registered new tserver with Master: bd929442052b4ca2b946cbb3ce3110c7 (127.8.13.3:39029)
I20260928 12:30:57.123916 11297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:52251
I20260928 12:30:57.125026 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:30:57 dist-test-slave-g31s krb5kdc[11252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598657, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 12:30:57 dist-test-slave-g31s krb5kdc[11252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598657, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:57.148234 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:57.140388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.8.13.62:39163)
0928 12:30:57.140444 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:57.140446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:57.140567 (+ 121us) server_negotiation.cc:407] Connection header received
0928 12:30:57.140683 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:57.140684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:57.140695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:57.140759 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:57.141439 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:57.141998 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:57.143085 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:57.143285 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:57.143350 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:57.145757 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:57.145760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:57.145760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:57.145767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:57.147155 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:57.147565 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:57.147566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:57.147566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:57.147629 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:57.147777 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:57.147777 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:57.147778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:57.147928 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:57.148008 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:57.148095 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 12:30:57.148164 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 12:30:57.148494 11297 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60928
I20260928 12:30:57.149024 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 11359
I20260928 12:30:57.154373 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 11493
I20260928 12:30:57.158972 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 11627
I20260928 12:30:57.162879 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 11265
2026-09-28T12:30:57Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1773 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:57 dist-test-slave-g31s krb5kdc[11768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:57 dist-test-slave-g31s krb5kdc[11768](info): set up 2 sockets
Sep 28 12:30:57 dist-test-slave-g31s krb5kdc[11768](info): commencing operation
krb5kdc: starting...
W20260928 12:30:58.129423 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:30:58 dist-test-slave-g31s krb5kdc[11768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
W20260928 12:30:59.188712 8244 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.13.62: real 0.952s user 0.001s sys 0.002s
I20260928 12:30:59.189076 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45113
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:45113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:59.296814 11781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:59.297065 11781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:59.297089 11781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:59.298471 11781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:59.298530 11781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:59.298545 11781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:59.298557 11781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:59.298568 11781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:59.300277 11781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:45113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:59.300565 11781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:59.300787 11781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:59.306532 11786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:59.306525 11789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:59.306694 11781 server_base.cc:1034] running on GCE node
W20260928 12:30:59.306532 11787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:59.307075 11781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:59.307405 11781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:59.308588 11781 hybrid_clock.cc:648] HybridClock initialized: now 1790598659308551 us; error 57 us; skew 500 ppm
Sep 28 12:30:59 dist-test-slave-g31s krb5kdc[11768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598659, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:59.311635 11781 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:30:59.312402 11781 webserver.cc:460] Webserver started at http://127.8.13.62:33195/ using document root <none> and password file <none>
I20260928 12:30:59.312659 11781 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:59.312712 11781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:59.312847 11781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:59.313997 11781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "cc10a4f0b58a48e3a30c2c471cfd8edd"
format_stamp: "Formatted at 2026-09-28 12:30:59 on dist-test-slave-g31s"
server_key: "321803be6baa2702b09c0aa6c67ff601"
server_key_iv: "277a517b69e6a61b347ddc7487a2d41c"
server_key_version: "encryptionkey@0"
I20260928 12:30:59.314360 11781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "cc10a4f0b58a48e3a30c2c471cfd8edd"
format_stamp: "Formatted at 2026-09-28 12:30:59 on dist-test-slave-g31s"
server_key: "321803be6baa2702b09c0aa6c67ff601"
server_key_iv: "277a517b69e6a61b347ddc7487a2d41c"
server_key_version: "encryptionkey@0"
I20260928 12:30:59.315750 11781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:59.316537 11796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:59.316742 11781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:59.316833 11781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "cc10a4f0b58a48e3a30c2c471cfd8edd"
format_stamp: "Formatted at 2026-09-28 12:30:59 on dist-test-slave-g31s"
server_key: "321803be6baa2702b09c0aa6c67ff601"
server_key_iv: "277a517b69e6a61b347ddc7487a2d41c"
server_key_version: "encryptionkey@0"
I20260928 12:30:59.316898 11781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:59.338757 11781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:59.341209 11781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:59.341365 11781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:59.345405 11781 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:45113
I20260928 12:30:59.345454 11848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:45113 every 8 connection(s)
I20260928 12:30:59.346001 11781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:59.347065 11849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:59.350623 11849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd: Bootstrap starting.
I20260928 12:30:59.351333 11849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:59.351665 11849 log.cc:824] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:59.352661 11849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd: No bootstrap required, opened a new log
I20260928 12:30:59.354521 11849 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10a4f0b58a48e3a30c2c471cfd8edd" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45113 } }
I20260928 12:30:59.354698 11849 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:59.354720 11849 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc10a4f0b58a48e3a30c2c471cfd8edd, State: Initialized, Role: FOLLOWER
I20260928 12:30:59.354861 11849 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10a4f0b58a48e3a30c2c471cfd8edd" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45113 } }
I20260928 12:30:59.354950 11849 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:59.354986 11849 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:59.355036 11849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:59.355351 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 11781
I20260928 12:30:59.355453 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:30:59.355624 8244 external_mini_cluster.cc:1428] Setting key 1832299441800d289ab6208cec55dc2b
Sep 28 12:30:59 dist-test-slave-g31s krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:30:59.367488 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:59.356343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.8.13.62:45113)
0928 12:30:59.356670 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:59.356703 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:59.356763 (+ 60us) server_negotiation.cc:407] Connection header received
0928 12:30:59.356899 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:59.356907 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:59.356961 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:59.357104 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:59.357890 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:59.358858 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:59.359952 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:59.360441 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:59.360549 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:59.362970 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:59.362974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:59.362974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:59.362990 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:59.365075 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:59.365685 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:59.365689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:59.365690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:59.365858 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:59.366180 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:59.366182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:59.366182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:59.366717 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:59.366874 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:59.367076 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 12:30:59.367249 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":121,"threads_started":1}
I20260928 12:30:59.384327 11849 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10a4f0b58a48e3a30c2c471cfd8edd" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45113 } }
I20260928 12:30:59.384616 11849 leader_election.cc:302] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc10a4f0b58a48e3a30c2c471cfd8edd; no voters:
I20260928 12:30:59.384969 11849 leader_election.cc:288] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:59.385076 11854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:59.385298 11854 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [term 1 LEADER]: Becoming Leader. State: Replica: cc10a4f0b58a48e3a30c2c471cfd8edd, State: Running, Role: LEADER
I20260928 12:30:59.385334 11849 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:59.385493 11854 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10a4f0b58a48e3a30c2c471cfd8edd" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45113 } }
I20260928 12:30:59.386009 11855 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc10a4f0b58a48e3a30c2c471cfd8edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10a4f0b58a48e3a30c2c471cfd8edd" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45113 } } }
I20260928 12:30:59.386102 11855 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:59.386339 11856 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc10a4f0b58a48e3a30c2c471cfd8edd. Latest consensus state: current_term: 1 leader_uuid: "cc10a4f0b58a48e3a30c2c471cfd8edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10a4f0b58a48e3a30c2c471cfd8edd" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45113 } } }
I20260928 12:30:59.386663 11856 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:59.386940 11859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:59.387634 11859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:59.389333 11859 catalog_manager.cc:1347] Generated new cluster ID: 1a12a9f97a554a39a9cb7fe017594cb5
I20260928 12:30:59.389379 11859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:59.405627 11859 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:59.406220 11859 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:59.409215 11859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc10a4f0b58a48e3a30c2c471cfd8edd: Generated new TSK 0
I20260928 12:30:59.409410 11859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
W20260928 12:31:00.512805 8244 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.13.1: real 1.097s user 0.000s sys 0.003s
I20260928 12:31:00.513252 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45113
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:00.599155 11875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:00.599390 11875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:00.599424 11875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:00.600708 11875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:00.600760 11875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:00.600800 11875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:00.602545 11875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:00.602887 11875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:00.603155 11875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:00.608937 11881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:00.608937 11883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:00.609165 11880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:00.609591 11875 server_base.cc:1034] running on GCE node
I20260928 12:31:00.609826 11875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:00.610093 11875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:00.611261 11875 hybrid_clock.cc:648] HybridClock initialized: now 1790598660611240 us; error 34 us; skew 500 ppm
Sep 28 12:31:00 dist-test-slave-g31s krb5kdc[11768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598660, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:00.614125 11875 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:31:00.614589 11875 webserver.cc:460] Webserver started at http://127.8.13.1:41131/ using document root <none> and password file <none>
I20260928 12:31:00.614790 11875 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:00.614837 11875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:00.614962 11875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:00.616003 11875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "dde94129f7334e56ad15de7723576763"
format_stamp: "Formatted at 2026-09-28 12:31:00 on dist-test-slave-g31s"
server_key: "edd14ed8d470ed297308a7e99c35eb8e"
server_key_iv: "540033e3002527775beb3bdf57997db7"
server_key_version: "encryptionkey@0"
I20260928 12:31:00.616334 11875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "dde94129f7334e56ad15de7723576763"
format_stamp: "Formatted at 2026-09-28 12:31:00 on dist-test-slave-g31s"
server_key: "edd14ed8d470ed297308a7e99c35eb8e"
server_key_iv: "540033e3002527775beb3bdf57997db7"
server_key_version: "encryptionkey@0"
I20260928 12:31:00.617641 11875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:00.618582 11890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:00.618847 11875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:00.618935 11875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "dde94129f7334e56ad15de7723576763"
format_stamp: "Formatted at 2026-09-28 12:31:00 on dist-test-slave-g31s"
server_key: "edd14ed8d470ed297308a7e99c35eb8e"
server_key_iv: "540033e3002527775beb3bdf57997db7"
server_key_version: "encryptionkey@0"
I20260928 12:31:00.619004 11875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:00.633320 11875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:00.635643 11875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:00.635798 11875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:00.636031 11875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:00.636334 11875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:00.636368 11875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:00.636399 11875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:00.636421 11875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:00.643457 11875 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:41963
I20260928 12:31:00.643600 12003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:41963 every 8 connection(s)
I20260928 12:31:00.643944 11875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:31:00.648773 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 11875
I20260928 12:31:00.648910 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:31:00.649112 8244 external_mini_cluster.cc:1428] Setting key c7fb64f2fe5ac70359228dc3b61fc1a4
Sep 28 12:31:00 dist-test-slave-g31s krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598660, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:00.656562 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:00.645313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:56865 (local address 127.8.13.62:45113)
0928 12:31:00.645447 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:00.645450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:00.645537 (+ 87us) server_negotiation.cc:407] Connection header received
0928 12:31:00.646767 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:00.646769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:00.646787 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:00.646916 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:00.647938 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:00.648496 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:00.650371 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:00.650608 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:00.650679 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:00.653720 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:00.653722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:00.653722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:00.653731 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:00.655186 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:00.655618 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:00.655619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:00.655619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:00.655677 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:00.655901 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:00.655902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:00.655902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:00.656030 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:00.656086 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:00.656367 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 12:31:00.656454 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260928 12:31:00.656824 12006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:00.645166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45113 (local address 127.8.13.1:56865)
0928 12:31:00.645383 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:00.645437 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:00.646608 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:00.647036 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:00.647042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:00.647411 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:00.647766 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:00.647788 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:00.648647 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:00.648648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:00.650200 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:00.650206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:00.650706 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:00.650707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:00.650879 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:00.651596 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:00.651608 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:00.653596 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:00.655275 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:00.655285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:00.655288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:00.655545 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:00.655734 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:00.655735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:00.655735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:00.655845 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:00.656087 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:00.656095 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:00.656247 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 12:31:00.656340 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 12:31:00.656536 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
I20260928 12:31:00.657436 12004 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45113
I20260928 12:31:00.657541 12004 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
I20260928 12:31:00.657752 12004 heartbeater.cc:511] Master 127.8.13.62:45113 requested a full tablet report, sending...
I20260928 12:31:00.658390 11813 ts_manager.cc:194] Registered new tserver with Master: dde94129f7334e56ad15de7723576763 (127.8.13.1:41963)
I20260928 12:31:00.659065 11813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:56865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
I20260928 12:31:01.660050 12004 heartbeater.cc:503] Master 127.8.13.62:45113 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
W20260928 12:31:01.732306 8244 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.13.2: real 1.082s user 0.000s sys 0.003s
I20260928 12:31:01.732739 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45113
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:01.821107 12010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:01.821385 12010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:01.821426 12010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:01.822865 12010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:01.822932 12010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:01.822973 12010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:01.824978 12010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:01.825243 12010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:01.825513 12010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:01.830799 12018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:01.830780 12015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:01.830780 12016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:01.831149 12010 server_base.cc:1034] running on GCE node
I20260928 12:31:01.831390 12010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:01.831633 12010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:01.832809 12010 hybrid_clock.cc:648] HybridClock initialized: now 1790598661832790 us; error 34 us; skew 500 ppm
Sep 28 12:31:01 dist-test-slave-g31s krb5kdc[11768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598661, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:01.839610 12010 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:31:01.840095 12010 webserver.cc:460] Webserver started at http://127.8.13.2:43911/ using document root <none> and password file <none>
I20260928 12:31:01.840384 12010 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:01.840466 12010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:01.840587 12010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:01.841723 12010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "6acff0f5e1204ea9be7b182d474f25c1"
format_stamp: "Formatted at 2026-09-28 12:31:01 on dist-test-slave-g31s"
server_key: "c1a489af7cf9afd9ba9824520984eaba"
server_key_iv: "166c9482f66514907b64c2ec503caa45"
server_key_version: "encryptionkey@0"
I20260928 12:31:01.842149 12010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "6acff0f5e1204ea9be7b182d474f25c1"
format_stamp: "Formatted at 2026-09-28 12:31:01 on dist-test-slave-g31s"
server_key: "c1a489af7cf9afd9ba9824520984eaba"
server_key_iv: "166c9482f66514907b64c2ec503caa45"
server_key_version: "encryptionkey@0"
I20260928 12:31:01.843562 12010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:01.844596 12025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:01.844796 12010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:01.844878 12010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "6acff0f5e1204ea9be7b182d474f25c1"
format_stamp: "Formatted at 2026-09-28 12:31:01 on dist-test-slave-g31s"
server_key: "c1a489af7cf9afd9ba9824520984eaba"
server_key_iv: "166c9482f66514907b64c2ec503caa45"
server_key_version: "encryptionkey@0"
I20260928 12:31:01.844964 12010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:01.864780 12010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:01.867640 12010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:01.867868 12010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:01.868177 12010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:01.868602 12010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:01.868647 12010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:01.868678 12010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:01.868721 12010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:01.876029 12010 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:40909
I20260928 12:31:01.876082 12138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:40909 every 8 connection(s)
I20260928 12:31:01.876590 12010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:31:01.879252 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 12010
I20260928 12:31:01.879362 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:31:01.879539 8244 external_mini_cluster.cc:1428] Setting key eb8ea38556d385f390b20e7823aec090
Sep 28 12:31:01 dist-test-slave-g31s krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598661, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
I20260928 12:31:01.891981 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:01.877446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:37409 (local address 127.8.13.62:45113)
0928 12:31:01.877572 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:01.877574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:01.877738 (+ 164us) server_negotiation.cc:407] Connection header received
0928 12:31:01.878955 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:01.878957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:01.878972 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:01.879038 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:01.880039 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:01.882234 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:01.883590 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:01.883867 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:01.883968 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:01.888291 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:01.888294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:01.888294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:01.888304 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:01.890164 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:01.890773 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:01.890774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:01.890774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:01.890881 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:01.891209 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:01.891211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:01.891211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:01.891343 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:01.891404 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:01.891770 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 12:31:01.891873 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260928 12:31:01.892293 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:01.877452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45113 (local address 127.8.13.2:37409)
0928 12:31:01.877608 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:01.877656 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:01.878813 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:01.879170 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:01.879177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:01.879591 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:01.879935 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:01.879944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:01.882465 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:01.882466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:01.883490 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:01.883494 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:01.883991 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:01.883992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:01.884157 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:01.885207 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:01.885220 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:01.888133 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:01.890272 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:01.890289 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:01.890292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:01.890691 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:01.890974 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:01.890976 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:01.890976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:01.891132 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:01.891419 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:01.891427 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:01.891617 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 12:31:01.891827 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 12:31:01.892043 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260928 12:31:01.892807 12139 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45113
I20260928 12:31:01.892935 12139 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:01.893180 12139 heartbeater.cc:511] Master 127.8.13.62:45113 requested a full tablet report, sending...
I20260928 12:31:01.893675 11813 ts_manager.cc:194] Registered new tserver with Master: 6acff0f5e1204ea9be7b182d474f25c1 (127.8.13.2:40909)
I20260928 12:31:01.894129 11813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:37409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
W20260928 12:31:02.735526 8244 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.13.3: real 0.855s user 0.001s sys 0.002s
I20260928 12:31:02.735972 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45113
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:02.821094 12145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:02.821298 12145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:02.821322 12145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:02.822594 12145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:02.822642 12145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:02.822671 12145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:02.824211 12145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:02.824425 12145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:02.824636 12145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:02.829273 12153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:02.829404 12145 server_base.cc:1034] running on GCE node
W20260928 12:31:02.829322 12151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:02.829274 12150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:02.829731 12145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:02.829983 12145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:02.831115 12145 hybrid_clock.cc:648] HybridClock initialized: now 1790598662831097 us; error 35 us; skew 500 ppm
Sep 28 12:31:02 dist-test-slave-g31s krb5kdc[11768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598662, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:02.839979 12145 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:31:02.840461 12145 webserver.cc:460] Webserver started at http://127.8.13.3:32955/ using document root <none> and password file <none>
I20260928 12:31:02.840693 12145 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:02.840745 12145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:02.840874 12145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:02.842123 12145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "efd18bab17bf4d69a2b00760cf25131f"
format_stamp: "Formatted at 2026-09-28 12:31:02 on dist-test-slave-g31s"
server_key: "f6107bc4c475a94a345ca4dd6b950008"
server_key_iv: "b30d61196b29ba51a981401a4913b987"
server_key_version: "encryptionkey@0"
I20260928 12:31:02.842499 12145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "efd18bab17bf4d69a2b00760cf25131f"
format_stamp: "Formatted at 2026-09-28 12:31:02 on dist-test-slave-g31s"
server_key: "f6107bc4c475a94a345ca4dd6b950008"
server_key_iv: "b30d61196b29ba51a981401a4913b987"
server_key_version: "encryptionkey@0"
I20260928 12:31:02.843954 12145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:02.844772 12160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:02.844970 12145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:02.845059 12145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "efd18bab17bf4d69a2b00760cf25131f"
format_stamp: "Formatted at 2026-09-28 12:31:02 on dist-test-slave-g31s"
server_key: "f6107bc4c475a94a345ca4dd6b950008"
server_key_iv: "b30d61196b29ba51a981401a4913b987"
server_key_version: "encryptionkey@0"
I20260928 12:31:02.845129 12145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:02.866086 12145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:02.868889 12145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:02.869063 12145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:02.869321 12145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:02.869660 12145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:02.869696 12145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:02.869728 12145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:02.869751 12145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:02.875540 12145 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:36673
I20260928 12:31:02.875588 12273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:36673 every 8 connection(s)
I20260928 12:31:02.875965 12145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:31:02.881350 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 12145
I20260928 12:31:02.881453 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:31:02.881629 8244 external_mini_cluster.cc:1428] Setting key dc3a51eeee5f83601e768ef741bf2a22
Sep 28 12:31:02 dist-test-slave-g31s krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598662, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:02.887439 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:02.876734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:52771 (local address 127.8.13.62:45113)
0928 12:31:02.876864 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:02.876866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:02.877010 (+ 144us) server_negotiation.cc:407] Connection header received
0928 12:31:02.878095 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:02.878096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:02.878112 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:02.878178 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:02.878916 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:02.879426 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:02.880774 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:02.880998 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:02.881074 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:02.883866 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:02.883868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:02.883868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:02.883876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:02.885412 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:02.886025 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:02.886026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:02.886027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:02.886092 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:02.886408 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:02.886409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:02.886410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:02.886560 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:02.886621 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:02.887256 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 12:31:02.887342 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260928 12:31:02.887645 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:02.876706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45113 (local address 127.8.13.3:52771)
0928 12:31:02.876872 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:02.876916 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:02.877960 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:02.878239 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:02.878242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:02.878508 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:02.878718 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:02.878723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:02.879597 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:02.879599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:02.880626 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:02.880629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:02.881153 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:02.881155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:02.881299 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:02.882114 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:02.882125 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:02.883729 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:02.885571 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:02.885580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:02.885582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:02.885938 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:02.886186 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:02.886187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:02.886187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:02.886307 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:02.886721 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:02.886731 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:02.886921 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 12:31:02.887212 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 12:31:02.887415 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260928 12:31:02.888139 12274 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45113
I20260928 12:31:02.888236 12274 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:02.888427 12274 heartbeater.cc:511] Master 127.8.13.62:45113 requested a full tablet report, sending...
I20260928 12:31:02.888787 11813 ts_manager.cc:194] Registered new tserver with Master: efd18bab17bf4d69a2b00760cf25131f (127.8.13.3:36673)
I20260928 12:31:02.889150 11813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:52771
I20260928 12:31:02.893872 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:31:02.895274 12139 heartbeater.cc:503] Master 127.8.13.62:45113 was elected leader, sending a full tablet report...
Sep 28 12:31:02 dist-test-slave-g31s krb5kdc[11768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:31:02 dist-test-slave-g31s krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:31:02.917304 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:02.909961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.8.13.1:41963)
0928 12:31:02.910109 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:02.910115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:02.910128 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:31:02.910158 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:02.910160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:02.910174 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:02.910240 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:02.910813 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:02.911534 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:02.912291 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:02.912620 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:02.912673 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:02.914867 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:02.914875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:02.914876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:02.914882 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:02.916313 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:02.916634 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:02.916636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:02.916636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:02.916687 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:02.916818 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:02.916818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:02.916819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:02.916939 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:02.917070 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:02.917104 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 12:31:02.917169 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
Sep 28 12:31:02 dist-test-slave-g31s krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:02.924482 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:02.918457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.8.13.62:45113)
0928 12:31:02.918520 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:02.918522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:02.918531 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:02.918595 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:02.918596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:02.918606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:02.918657 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:02.919072 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:02.919560 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:02.920203 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:02.920406 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:02.920458 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:02.922497 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:02.922498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:02.922498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:02.922505 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:02.923599 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:02.923879 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:02.923880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:02.923880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:02.923925 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:02.924057 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:02.924057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:02.924058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:02.924193 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:02.924267 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:02.924337 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 12:31:02.924403 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 12:31:02.924726 11813 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54342
I20260928 12:31:02.925062 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 11875
I20260928 12:31:02.929558 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 12010
I20260928 12:31:02.933413 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 12145
I20260928 12:31:02.937316 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 11781
2026-09-28T12:31:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5774 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:31:02 dist-test-slave-g31s krb5kdc[12287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:31:02 dist-test-slave-g31s krb5kdc[12287](info): set up 2 sockets
Sep 28 12:31:02 dist-test-slave-g31s krb5kdc[12287](info): commencing operation
krb5kdc: starting...
W20260928 12:31:03.905753 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:31:03 dist-test-slave-g31s krb5kdc[12287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:31:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:31:04.026304 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:41259
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:41259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:04.110838 12300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:04.111063 12300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:04.111097 12300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:04.112325 12300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:04.112380 12300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:04.112397 12300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:04.112413 12300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:04.112430 12300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:04.114136 12300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:41259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:41259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:04.114403 12300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:04.114627 12300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:04.119488 12308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:04.119776 12300 server_base.cc:1034] running on GCE node
W20260928 12:31:04.119519 12306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:04.120373 12305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:04.120644 12300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:04.120959 12300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:04.122112 12300 hybrid_clock.cc:648] HybridClock initialized: now 1790598664122095 us; error 38 us; skew 500 ppm
Sep 28 12:31:04 dist-test-slave-g31s krb5kdc[12287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598664, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:04.124720 12300 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:31:04.125048 12300 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:04.125097 12300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:04.125206 12300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:04.126284 12300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "75efc9094b6244bf905434caf957365f"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "ad03b0698ee64c456c7eafb22dd5d850"
server_key_iv: "c8bdf2d568035915be283f1341ff1cfb"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.126624 12300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "75efc9094b6244bf905434caf957365f"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "ad03b0698ee64c456c7eafb22dd5d850"
server_key_iv: "c8bdf2d568035915be283f1341ff1cfb"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.128293 12300 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 12:31:04.129422 12314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:04.129648 12300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:04.129741 12300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "75efc9094b6244bf905434caf957365f"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "ad03b0698ee64c456c7eafb22dd5d850"
server_key_iv: "c8bdf2d568035915be283f1341ff1cfb"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.129832 12300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:04.153065 12300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:04.155417 12300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:04.155581 12300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:04.160533 12300 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:41259
I20260928 12:31:04.160635 12366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:41259 every 8 connection(s)
I20260928 12:31:04.161088 12300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:04.162185 12367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:04.164894 12367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f: Bootstrap starting.
I20260928 12:31:04.165491 12367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:04.165755 12367 log.cc:824] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:04.166473 12367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f: No bootstrap required, opened a new log
I20260928 12:31:04.168071 12367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75efc9094b6244bf905434caf957365f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41259 } }
I20260928 12:31:04.168221 12367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:04.168242 12367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75efc9094b6244bf905434caf957365f, State: Initialized, Role: FOLLOWER
I20260928 12:31:04.168360 12367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75efc9094b6244bf905434caf957365f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41259 } }
I20260928 12:31:04.168435 12367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:04.168457 12367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:04.168490 12367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:04.169329 12367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75efc9094b6244bf905434caf957365f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41259 } }
I20260928 12:31:04.169423 12367 leader_election.cc:302] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75efc9094b6244bf905434caf957365f; no voters:
I20260928 12:31:04.169567 12367 leader_election.cc:288] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:04.169646 12370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:04.169836 12367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:04.169915 12370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [term 1 LEADER]: Becoming Leader. State: Replica: 75efc9094b6244bf905434caf957365f, State: Running, Role: LEADER
I20260928 12:31:04.170013 12370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75efc9094b6244bf905434caf957365f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41259 } }
I20260928 12:31:04.170388 12371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75efc9094b6244bf905434caf957365f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75efc9094b6244bf905434caf957365f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41259 } } }
I20260928 12:31:04.170372 12372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75efc9094b6244bf905434caf957365f. Latest consensus state: current_term: 1 leader_uuid: "75efc9094b6244bf905434caf957365f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75efc9094b6244bf905434caf957365f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41259 } } }
I20260928 12:31:04.170526 12371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:04.170532 12372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:04.170742 12379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:04.171240 12379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:04.171336 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 12300
I20260928 12:31:04.171423 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:04.171622 8244 external_mini_cluster.cc:1428] Setting key 87299a43a4cc666f4654859807fff27a
I20260928 12:31:04.172879 12379 catalog_manager.cc:1347] Generated new cluster ID: 8e1294e5c7bf45a8a59788f8ca3cae32
I20260928 12:31:04.172931 12379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:31:04 dist-test-slave-g31s krb5kdc[12287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:04.181036 12379 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:04.181938 12379 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:04.186451 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.172059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.8.13.62:41259)
0928 12:31:04.172204 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:04.172214 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:04.172249 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:31:04.172546 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:04.172554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:04.172622 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:04.172758 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:04.173896 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.175287 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.176431 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.177154 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.177259 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.180337 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:04.180345 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:04.180345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:04.180362 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:04.183287 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:04.183988 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:04.183992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:04.183992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:04.184184 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:04.184525 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:04.184525 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:04.184526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:04.184972 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:04.185721 (+ 749us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:04.185849 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 12:31:04.186099 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260928 12:31:04.193665 12379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f: Generated new TSK 0
I20260928 12:31:04.193918 12379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:31:04.225097 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41259
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:04.310735 12393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:04.310983 12393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:04.311012 12393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:04.312340 12393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:04.312400 12393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:04.312440 12393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:04.314162 12393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:04.314431 12393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:04.314674 12393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:04.320385 12401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:04.320385 12398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:04.320465 12393 server_base.cc:1034] running on GCE node
W20260928 12:31:04.320405 12399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:04.320883 12393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:04.321120 12393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:04.322261 12393 hybrid_clock.cc:648] HybridClock initialized: now 1790598664322248 us; error 28 us; skew 500 ppm
Sep 28 12:31:04 dist-test-slave-g31s krb5kdc[12287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598664, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:04.324879 12393 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:31:04.325208 12393 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:04.325256 12393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:04.325376 12393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:04.326496 12393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "ccefc8f914dc4cbfa6927e1311291419"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "5cd07f8dc76f3170dab96cddf546621a"
server_key_iv: "480286a128987077adcd0a80440db6e5"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.326829 12393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "ccefc8f914dc4cbfa6927e1311291419"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "5cd07f8dc76f3170dab96cddf546621a"
server_key_iv: "480286a128987077adcd0a80440db6e5"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.328078 12393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:04.328785 12407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:04.328953 12393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:31:04.329054 12393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "ccefc8f914dc4cbfa6927e1311291419"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "5cd07f8dc76f3170dab96cddf546621a"
server_key_iv: "480286a128987077adcd0a80440db6e5"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.329128 12393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:04.350068 12393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:04.352865 12393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:04.353012 12393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:04.353286 12393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:04.353637 12393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:04.353673 12393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:04.353706 12393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:04.353729 12393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:04.359493 12393 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:38647
I20260928 12:31:04.359551 12520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:38647 every 8 connection(s)
I20260928 12:31:04.359813 12393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:31:04.360695 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 12393
I20260928 12:31:04.360808 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:31:04.361022 8244 external_mini_cluster.cc:1428] Setting key 76fa55a7ed451b5af09346f7df6c4830
Sep 28 12:31:04 dist-test-slave-g31s krb5kdc[12287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598664, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
I20260928 12:31:04.371719 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.360695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:47473 (local address 127.8.13.62:41259)
0928 12:31:04.360846 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:04.360849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:04.361017 (+ 168us) server_negotiation.cc:407] Connection header received
0928 12:31:04.362026 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:04.362027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:04.362040 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:04.362083 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:04.363198 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.363723 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.365124 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.365421 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.365500 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.368243 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:04.368247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:04.368247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:04.368259 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:04.370000 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:04.370536 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:04.370538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:04.370538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:04.370600 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:04.370850 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:04.370851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:04.370851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:04.371012 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:04.371085 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:04.371566 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 12:31:04.371648 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 12:31:04.372074 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.360545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:41259 (local address 127.8.13.1:47473)
0928 12:31:04.360807 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:04.360895 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:04.361905 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:04.362171 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:04.362175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:04.362490 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:04.363079 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.363087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.363829 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.363830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.364965 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.364972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.365546 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.365548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.365716 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.366613 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:04.366624 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:04.368100 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:04.370098 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:04.370108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:04.370111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:04.370451 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:04.370668 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:04.370668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:04.370668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:04.370784 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:04.371119 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:04.371136 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:04.371386 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 12:31:04.371533 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 12:31:04.371786 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":106,"threads_started":1}
I20260928 12:31:04.372758 12521 heartbeater.cc:347] Connected to a master server at 127.8.13.62:41259
I20260928 12:31:04.372960 12521 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:04.373178 12521 heartbeater.cc:511] Master 127.8.13.62:41259 requested a full tablet report, sending...
I20260928 12:31:04.374053 12330 ts_manager.cc:194] Registered new tserver with Master: ccefc8f914dc4cbfa6927e1311291419 (127.8.13.1:38647)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
I20260928 12:31:04.375156 12330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:47473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:31:04.389467 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41259
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:04.474591 12526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:04.474852 12526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:04.474891 12526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:04.476241 12526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:04.476301 12526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:04.476343 12526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:04.478241 12526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:04.478523 12526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:04.478772 12526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:04.484037 12532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:04.484047 12534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:04.484165 12526 server_base.cc:1034] running on GCE node
W20260928 12:31:04.484051 12531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:04.484558 12526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:04.484840 12526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:04.486076 12526 hybrid_clock.cc:648] HybridClock initialized: now 1790598664486057 us; error 31 us; skew 500 ppm
Sep 28 12:31:04 dist-test-slave-g31s krb5kdc[12287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598664, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:04.489123 12526 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:31:04.489523 12526 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:04.489579 12526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:04.489710 12526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:04.490851 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "6466a4943b004b9eaf3a53c1af65fb34"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "a8567e86cd639dd8870f3c7a97639228"
server_key_iv: "5b9d288b0f6fd14ddd90fb82f4def2c3"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.491214 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "6466a4943b004b9eaf3a53c1af65fb34"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "a8567e86cd639dd8870f3c7a97639228"
server_key_iv: "5b9d288b0f6fd14ddd90fb82f4def2c3"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.492515 12526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:31:04.493438 12540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:04.493781 12526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:04.493973 12526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "6466a4943b004b9eaf3a53c1af65fb34"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "a8567e86cd639dd8870f3c7a97639228"
server_key_iv: "5b9d288b0f6fd14ddd90fb82f4def2c3"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.494076 12526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:04.522233 12526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:04.523990 12526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:04.524121 12526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:04.524328 12526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:04.524622 12526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:04.524653 12526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:04.524682 12526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:04.524700 12526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:04.529963 12526 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:40537
I20260928 12:31:04.530002 12653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:40537 every 8 connection(s)
I20260928 12:31:04.530248 12526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:31:04.535771 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 12526
I20260928 12:31:04.535883 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:31:04.536104 8244 external_mini_cluster.cc:1428] Setting key 827c54ace749b7f2ad251650bd49b802
Sep 28 12:31:04 dist-test-slave-g31s krb5kdc[12287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598664, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:04.539733 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.530903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:38341 (local address 127.8.13.62:41259)
0928 12:31:04.530978 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:04.530980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:04.531123 (+ 143us) server_negotiation.cc:407] Connection header received
0928 12:31:04.531626 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:04.531627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:04.531639 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:04.531688 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:04.532718 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.533277 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.534299 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.534494 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.534549 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.536658 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:04.536661 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:04.536661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:04.536672 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:04.538415 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:04.538853 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:04.538855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:04.538855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:04.538913 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:04.539137 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:04.539138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:04.539138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:04.539250 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:04.539307 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:04.539573 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 12:31:04.539642 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 12:31:04.539965 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.530865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:41259 (local address 127.8.13.2:38341)
0928 12:31:04.531005 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:04.531046 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:04.531535 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:04.531754 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:04.531758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:04.532103 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:04.532635 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.532641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.533361 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.533362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.534224 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.534227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.534561 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.534562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.534675 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.535139 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:04.535147 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:04.536552 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:04.538515 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:04.538525 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:04.538528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:04.538778 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:04.538976 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:04.538976 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:04.538977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:04.539081 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:04.539305 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:04.539312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:04.539451 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 12:31:04.539551 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 12:31:04.539721 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260928 12:31:04.540462 12654 heartbeater.cc:347] Connected to a master server at 127.8.13.62:41259
I20260928 12:31:04.540582 12654 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:04.540735 12654 heartbeater.cc:511] Master 127.8.13.62:41259 requested a full tablet report, sending...
I20260928 12:31:04.541117 12330 ts_manager.cc:194] Registered new tserver with Master: 6466a4943b004b9eaf3a53c1af65fb34 (127.8.13.2:40537)
I20260928 12:31:04.541481 12330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:38341
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:31:04.561328 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41259
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:04.644448 12659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:04.644681 12659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:04.644716 12659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:04.646010 12659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:04.646059 12659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:04.646100 12659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:04.647764 12659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:04.648032 12659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:04.648252 12659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:04.653318 12664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:04.653616 12665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:04.653592 12667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:04.654017 12659 server_base.cc:1034] running on GCE node
I20260928 12:31:04.654255 12659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:04.654500 12659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:04.655656 12659 hybrid_clock.cc:648] HybridClock initialized: now 1790598664655642 us; error 29 us; skew 500 ppm
Sep 28 12:31:04 dist-test-slave-g31s krb5kdc[12287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598664, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:04.658356 12659 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:31:04.658723 12659 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:04.658775 12659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:04.658907 12659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:04.660003 12659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "b5affc5b55fa46ffb39b152bb8443314"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "1bdd8c4840fc3f16dfc14e49fa4fd33d"
server_key_iv: "b5f1a26765843e57470e2db239bb5e49"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.660346 12659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "b5affc5b55fa46ffb39b152bb8443314"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "1bdd8c4840fc3f16dfc14e49fa4fd33d"
server_key_iv: "b5f1a26765843e57470e2db239bb5e49"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.661789 12659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:31:04.662655 12673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:04.662868 12659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:04.662957 12659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "b5affc5b55fa46ffb39b152bb8443314"
format_stamp: "Formatted at 2026-09-28 12:31:04 on dist-test-slave-g31s"
server_key: "1bdd8c4840fc3f16dfc14e49fa4fd33d"
server_key_iv: "b5f1a26765843e57470e2db239bb5e49"
server_key_version: "encryptionkey@0"
I20260928 12:31:04.663039 12659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:04.676685 12659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:04.679368 12659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:04.679530 12659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:04.679781 12659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:04.680106 12659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:04.680140 12659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:04.680163 12659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:04.680186 12659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:04.686817 12659 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:43647
I20260928 12:31:04.686921 12786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:43647 every 8 connection(s)
I20260928 12:31:04.687212 12659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:31:04.687630 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 12659
I20260928 12:31:04.687717 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:31:04.687896 8244 external_mini_cluster.cc:1428] Setting key 31f7a6626ad6153cf5eb6463d065f917
Sep 28 12:31:04 dist-test-slave-g31s krb5kdc[12287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598664, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:04.697203 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.688467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:58571 (local address 127.8.13.62:41259)
0928 12:31:04.688515 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:04.688517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:04.688774 (+ 257us) server_negotiation.cc:407] Connection header received
0928 12:31:04.689299 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:04.689300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:04.689310 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:04.689392 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:04.690383 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.690925 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.691942 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.692137 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.692193 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.694546 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:04.694548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:04.694549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:04.694556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:04.695848 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:04.696243 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:04.696244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:04.696244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:04.696303 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:04.696521 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:04.696523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:04.696523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:04.696647 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:04.696712 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:04.697050 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 12:31:04.697132 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:31:04.697395 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.688512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:41259 (local address 127.8.13.3:58571)
0928 12:31:04.688672 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:04.688708 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:04.689195 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:04.689445 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:04.689449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:04.689738 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:04.690277 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.690282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.691023 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.691024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.691853 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.691855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.692237 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.692238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.692372 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.692887 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:04.692894 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:04.694448 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:04.695931 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:04.695937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:04.695939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:04.696166 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:04.696354 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:04.696355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:04.696355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:04.696448 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:04.696740 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:04.696746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:04.696887 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 12:31:04.696987 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 12:31:04.697152 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260928 12:31:04.697837 12787 heartbeater.cc:347] Connected to a master server at 127.8.13.62:41259
I20260928 12:31:04.697933 12787 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:04.698078 12787 heartbeater.cc:511] Master 127.8.13.62:41259 requested a full tablet report, sending...
I20260928 12:31:04.698469 12330 ts_manager.cc:194] Registered new tserver with Master: b5affc5b55fa46ffb39b152bb8443314 (127.8.13.3:43647)
I20260928 12:31:04.698845 12330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:58571
I20260928 12:31:04.700042 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:31:04.708199 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.701259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.8.13.62:41259)
0928 12:31:04.701331 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:04.701333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:04.701404 (+ 71us) server_negotiation.cc:407] Connection header received
0928 12:31:04.701467 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:04.701468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:04.701477 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:04.701524 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:04.702129 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.702619 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.703329 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.703520 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.703599 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.705104 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:04.705107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:04.705107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:04.705117 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:04.706723 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:04.707317 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:04.707319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:04.707319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:04.707400 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:04.707591 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:04.707591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:04.707592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:04.707771 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:04.707875 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:04.707991 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 12:31:04.708097 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 12:31:04.709758 12330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:31:04.711479 12330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:31:04.717787 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.713749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:38647 (local address 127.0.0.1:38228)
0928 12:31:04.714289 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:04.714297 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:04.714346 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:04.714564 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:04.714565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:04.714567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:04.714596 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.714597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.715883 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.715884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.716814 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.716816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.717494 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.717495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.717571 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.717585 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:31:04.717650 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:31:04.717704 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":2,"thread_start_us":119,"threads_started":1}
I20260928 12:31:04.717830 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.714013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.8.13.1:38647)
0928 12:31:04.714197 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:04.714201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:04.714358 (+ 157us) server_negotiation.cc:407] Connection header received
0928 12:31:04.714406 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:04.714408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:04.714423 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:04.714490 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:04.714664 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.715779 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.716880 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.717414 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.717480 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.717550 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:04.717669 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 12:31:04.717709 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260928 12:31:04.718040 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.713735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:43647 (local address 127.0.0.1:38874)
0928 12:31:04.713900 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:04.713940 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:04.714009 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:04.714500 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:04.714500 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:04.714503 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:04.714563 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.714565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.715488 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.715489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.716816 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.716818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.717762 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.717763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.717853 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.717856 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:04.717901 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:31:04.717957 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260928 12:31:04.718197 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.713736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:40537 (local address 127.0.0.1:37320)
0928 12:31:04.714162 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:04.714174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:04.714227 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:04.714464 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:04.714467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:04.714469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:04.714573 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.714575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.715875 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.715876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.716937 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:04.716939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.717855 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.717856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:04.717920 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.717923 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:04.717962 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 12:31:04.717993 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":36,"thread_start_us":109,"threads_started":1}
I20260928 12:31:04.718199 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.714020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.8.13.2:40537)
0928 12:31:04.714323 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:04.714326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:04.714334 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:04.714367 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:04.714369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:04.714382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:04.714451 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:04.714655 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.715766 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.717016 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.717702 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.717753 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.717826 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:04.718080 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 12:31:04.718125 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":71,"threads_started":1}
I20260928 12:31:04.718568 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:04.713994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38874 (local address 127.8.13.3:43647)
0928 12:31:04.714263 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:04.714267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:04.714276 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:04.714316 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:04.714318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:04.714334 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:04.714407 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:04.714637 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.715391 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.716900 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:04.717608 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:04.717672 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:04.717727 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:04.718477 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 12:31:04.718512 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":92,"threads_started":1}
I20260928 12:31:04.718853 12455 tablet_service.cc:1467] Processing CreateTablet for tablet cfcf39b1be5045ebb8beca0c6175a2b4 (DEFAULT_TABLE table=test-table [id=89d6b5327d8f457d8cfbb6d988092425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:04.718853 12588 tablet_service.cc:1467] Processing CreateTablet for tablet cfcf39b1be5045ebb8beca0c6175a2b4 (DEFAULT_TABLE table=test-table [id=89d6b5327d8f457d8cfbb6d988092425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:04.719120 12721 tablet_service.cc:1467] Processing CreateTablet for tablet cfcf39b1be5045ebb8beca0c6175a2b4 (DEFAULT_TABLE table=test-table [id=89d6b5327d8f457d8cfbb6d988092425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:04.719213 12588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcf39b1be5045ebb8beca0c6175a2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:04.719213 12455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcf39b1be5045ebb8beca0c6175a2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:04.719345 12721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcf39b1be5045ebb8beca0c6175a2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:04.721815 12804 tablet_bootstrap.cc:492] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: Bootstrap starting.
I20260928 12:31:04.722368 12804 tablet_bootstrap.cc:654] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:04.722414 12806 tablet_bootstrap.cc:492] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: Bootstrap starting.
I20260928 12:31:04.722939 12804 log.cc:824] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:04.723023 12806 tablet_bootstrap.cc:654] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:04.723304 12805 tablet_bootstrap.cc:492] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419: Bootstrap starting.
I20260928 12:31:04.723549 12806 log.cc:824] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:04.723675 12804 tablet_bootstrap.cc:492] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: No bootstrap required, opened a new log
I20260928 12:31:04.723747 12804 ts_tablet_manager.cc:1397] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:31:04.723918 12805 tablet_bootstrap.cc:654] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:04.724385 12806 tablet_bootstrap.cc:492] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: No bootstrap required, opened a new log
I20260928 12:31:04.724447 12805 log.cc:824] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:04.724488 12806 ts_tablet_manager.cc:1397] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:31:04.725256 12805 tablet_bootstrap.cc:492] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419: No bootstrap required, opened a new log
I20260928 12:31:04.725325 12805 ts_tablet_manager.cc:1397] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:31:04.725684 12804 raft_consensus.cc:348] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6466a4943b004b9eaf3a53c1af65fb34" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 40537 } } peers { permanent_uuid: "b5affc5b55fa46ffb39b152bb8443314" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 43647 } } peers { permanent_uuid: "ccefc8f914dc4cbfa6927e1311291419" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38647 } }
I20260928 12:31:04.725857 12804 raft_consensus.cc:374] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:04.725894 12804 raft_consensus.cc:729] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5affc5b55fa46ffb39b152bb8443314, State: Initialized, Role: FOLLOWER
I20260928 12:31:04.726044 12804 consensus_queue.cc:260] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6466a4943b004b9eaf3a53c1af65fb34" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 40537 } } peers { permanent_uuid: "b5affc5b55fa46ffb39b152bb8443314" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 43647 } } peers { permanent_uuid: "ccefc8f914dc4cbfa6927e1311291419" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38647 } }
I20260928 12:31:04.726296 12804 ts_tablet_manager.cc:1428] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:31:04.726415 12787 heartbeater.cc:503] Master 127.8.13.62:41259 was elected leader, sending a full tablet report...
I20260928 12:31:04.726431 12806 raft_consensus.cc:348] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6466a4943b004b9eaf3a53c1af65fb34" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 40537 } } peers { permanent_uuid: "b5affc5b55fa46ffb39b152bb8443314" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 43647 } } peers { permanent_uuid: "ccefc8f914dc4cbfa6927e1311291419" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38647 } }
I20260928 12:31:04.726578 12806 raft_consensus.cc:374] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:04.726613 12806 raft_consensus.cc:729] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6466a4943b004b9eaf3a53c1af65fb34, State: Initialized, Role: FOLLOWER
I20260928 12:31:04.726718 12806 consensus_queue.cc:260] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6466a4943b004b9eaf3a53c1af65fb34" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 40537 } } peers { permanent_uuid: "b5affc5b55fa46ffb39b152bb8443314" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 43647 } } peers { permanent_uuid: "ccefc8f914dc4cbfa6927e1311291419" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38647 } }
I20260928 12:31:04.726974 12654 heartbeater.cc:503] Master 127.8.13.62:41259 was elected leader, sending a full tablet report...
I20260928 12:31:04.726985 12806 ts_tablet_manager.cc:1428] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:31:04.727829 12805 raft_consensus.cc:348] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6466a4943b004b9eaf3a53c1af65fb34" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 40537 } } peers { permanent_uuid: "b5affc5b55fa46ffb39b152bb8443314" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 43647 } } peers { permanent_uuid: "ccefc8f914dc4cbfa6927e1311291419" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38647 } }
I20260928 12:31:04.727978 12805 raft_consensus.cc:374] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:04.728001 12805 raft_consensus.cc:729] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccefc8f914dc4cbfa6927e1311291419, State: Initialized, Role: FOLLOWER
I20260928 12:31:04.728102 12805 consensus_queue.cc:260] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6466a4943b004b9eaf3a53c1af65fb34" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 40537 } } peers { permanent_uuid: "b5affc5b55fa46ffb39b152bb8443314" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 43647 } } peers { permanent_uuid: "ccefc8f914dc4cbfa6927e1311291419" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38647 } }
I20260928 12:31:04.728317 12805 ts_tablet_manager.cc:1428] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:31:04.728343 12521 heartbeater.cc:503] Master 127.8.13.62:41259 was elected leader, sending a full tablet report...
W20260928 12:31:04.780787 12655 tablet.cc:2369] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:31:04.860543 12522 tablet.cc:2369] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:31:04.937852 12788 tablet.cc:2369] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:31:05.036391 12812 raft_consensus.cc:482] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:05.036535 12812 raft_consensus.cc:504] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6466a4943b004b9eaf3a53c1af65fb34" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 40537 } } peers { permanent_uuid: "b5affc5b55fa46ffb39b152bb8443314" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 43647 } } peers { permanent_uuid: "ccefc8f914dc4cbfa6927e1311291419" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38647 } }
I20260928 12:31:05.036929 12812 leader_election.cc:288] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6466a4943b004b9eaf3a53c1af65fb34 (127.8.13.2:40537), b5affc5b55fa46ffb39b152bb8443314 (127.8.13.3:43647)
I20260928 12:31:05.040244 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.037126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:40537 (local address 127.8.13.1:51001)
0928 12:31:05.037293 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.037305 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:05.037398 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:05.037563 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:05.037564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:05.037568 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:05.037649 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:05.037656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.038318 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.038318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:05.039180 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:05.039183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.039921 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.039923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:05.040002 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:05.040008 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:31:05.040086 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 12:31:05.040135 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260928 12:31:05.040274 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.037246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:51001 (local address 127.8.13.2:40537)
0928 12:31:05.037324 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.037326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.037385 (+ 59us) server_negotiation.cc:407] Connection header received
0928 12:31:05.037481 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.037482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.037496 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.037551 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:05.037719 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.038236 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.039286 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.039792 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.039868 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:05.039880 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.040152 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.040196 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 12:31:05.040462 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.037274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:51659 (local address 127.8.13.3:43647)
0928 12:31:05.037333 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.037334 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.037460 (+ 126us) server_negotiation.cc:407] Connection header received
0928 12:31:05.037517 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.037517 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.037528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.037581 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:05.037738 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.038236 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.039357 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.040040 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.040126 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:05.040141 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.040336 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.040402 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:31:05.040675 12814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.037198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:43647 (local address 127.8.13.1:51659)
0928 12:31:05.037378 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.037388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:05.037464 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:05.037591 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:05.037592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:05.037594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:05.037671 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:05.037673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.038314 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.038316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:05.039182 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:05.039184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.040157 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.040158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:05.040232 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:05.040237 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:31:05.040551 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 12:31:05.040607 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260928 12:31:05.040632 12608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcf39b1be5045ebb8beca0c6175a2b4" candidate_uuid: "ccefc8f914dc4cbfa6927e1311291419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6466a4943b004b9eaf3a53c1af65fb34" is_pre_election: true
I20260928 12:31:05.040798 12608 raft_consensus.cc:2454] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccefc8f914dc4cbfa6927e1311291419 in term 0.
I20260928 12:31:05.041023 12741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcf39b1be5045ebb8beca0c6175a2b4" candidate_uuid: "ccefc8f914dc4cbfa6927e1311291419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5affc5b55fa46ffb39b152bb8443314" is_pre_election: true
I20260928 12:31:05.041147 12741 raft_consensus.cc:2454] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccefc8f914dc4cbfa6927e1311291419 in term 0.
I20260928 12:31:05.041374 12408 leader_election.cc:302] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6466a4943b004b9eaf3a53c1af65fb34, ccefc8f914dc4cbfa6927e1311291419; no voters:
I20260928 12:31:05.041700 12812 raft_consensus.cc:2781] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:31:05.041766 12812 raft_consensus.cc:482] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:31:05.041824 12812 raft_consensus.cc:3037] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:05.042550 12812 raft_consensus.cc:504] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6466a4943b004b9eaf3a53c1af65fb34" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 40537 } } peers { permanent_uuid: "b5affc5b55fa46ffb39b152bb8443314" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 43647 } } peers { permanent_uuid: "ccefc8f914dc4cbfa6927e1311291419" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38647 } }
I20260928 12:31:05.042934 12812 leader_election.cc:288] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [CANDIDATE]: Term 1 election: Requested vote from peers 6466a4943b004b9eaf3a53c1af65fb34 (127.8.13.2:40537), b5affc5b55fa46ffb39b152bb8443314 (127.8.13.3:43647)
I20260928 12:31:05.043102 12741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcf39b1be5045ebb8beca0c6175a2b4" candidate_uuid: "ccefc8f914dc4cbfa6927e1311291419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5affc5b55fa46ffb39b152bb8443314"
I20260928 12:31:05.043107 12608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcf39b1be5045ebb8beca0c6175a2b4" candidate_uuid: "ccefc8f914dc4cbfa6927e1311291419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6466a4943b004b9eaf3a53c1af65fb34"
I20260928 12:31:05.043184 12741 raft_consensus.cc:3037] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:05.043185 12608 raft_consensus.cc:3037] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:05.043893 12741 raft_consensus.cc:2454] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccefc8f914dc4cbfa6927e1311291419 in term 1.
I20260928 12:31:05.043891 12608 raft_consensus.cc:2454] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccefc8f914dc4cbfa6927e1311291419 in term 1.
I20260928 12:31:05.044061 12408 leader_election.cc:302] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6466a4943b004b9eaf3a53c1af65fb34, ccefc8f914dc4cbfa6927e1311291419; no voters:
I20260928 12:31:05.044261 12812 raft_consensus.cc:2781] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:05.044428 12812 raft_consensus.cc:686] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [term 1 LEADER]: Becoming Leader. State: Replica: ccefc8f914dc4cbfa6927e1311291419, State: Running, Role: LEADER
I20260928 12:31:05.044533 12812 consensus_queue.cc:237] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6466a4943b004b9eaf3a53c1af65fb34" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 40537 } } peers { permanent_uuid: "b5affc5b55fa46ffb39b152bb8443314" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 43647 } } peers { permanent_uuid: "ccefc8f914dc4cbfa6927e1311291419" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38647 } }
I20260928 12:31:05.045356 12331 catalog_manager.cc:5495] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccefc8f914dc4cbfa6927e1311291419 (127.8.13.1). New cstate: current_term: 1 leader_uuid: "ccefc8f914dc4cbfa6927e1311291419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6466a4943b004b9eaf3a53c1af65fb34" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 40537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5affc5b55fa46ffb39b152bb8443314" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccefc8f914dc4cbfa6927e1311291419" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38647 } health_report { overall_health: HEALTHY } } }
I20260928 12:31:05.075742 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.072846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.8.13.1:38647)
0928 12:31:05.072904 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.072906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.072915 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:05.072961 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.072962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.072981 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.073050 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:31:05.073342 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.074116 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.074930 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.075137 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.075198 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:05.075373 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:31:05.075485 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:31:05.075549 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.075655 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.075677 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:31:05.077941 12741 raft_consensus.cc:1261] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314 [term 1 FOLLOWER]: Refusing update from remote peer ccefc8f914dc4cbfa6927e1311291419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:05.077986 12608 raft_consensus.cc:1261] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34 [term 1 FOLLOWER]: Refusing update from remote peer ccefc8f914dc4cbfa6927e1311291419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:05.078337 12815 consensus_queue.cc:1035] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5affc5b55fa46ffb39b152bb8443314" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:05.078486 12812 consensus_queue.cc:1035] T cfcf39b1be5045ebb8beca0c6175a2b4 P ccefc8f914dc4cbfa6927e1311291419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6466a4943b004b9eaf3a53c1af65fb34" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 40537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:05.081429 12818 mvcc.cc:205] Tried to move back new op lower bound from 7334292132155109376 to 7334292132022968320. Current Snapshot: MvccSnapshot[applied={T|T < 7334292132155109376}]
I20260928 12:31:05.081970 12819 mvcc.cc:205] Tried to move back new op lower bound from 7334292132155109376 to 7334292132022968320. Current Snapshot: MvccSnapshot[applied={T|T < 7334292132155109376}]
I20260928 12:31:05.082850 12820 mvcc.cc:205] Tried to move back new op lower bound from 7334292132155109376 to 7334292132022968320. Current Snapshot: MvccSnapshot[applied={T|T < 7334292132155109376 or (T in {7334292132155109376})}]
I20260928 12:31:05.084566 12330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:05.084678 12330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:05.086037 12330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75efc9094b6244bf905434caf957365f: Sending DeleteTablet for 3 replicas of tablet cfcf39b1be5045ebb8beca0c6175a2b4
I20260928 12:31:05.086515 12455 tablet_service.cc:1514] Processing DeleteTablet for tablet cfcf39b1be5045ebb8beca0c6175a2b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:05 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:38228
I20260928 12:31:05.086658 12588 tablet_service.cc:1514] Processing DeleteTablet for tablet cfcf39b1be5045ebb8beca0c6175a2b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:05 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:37320
I20260928 12:31:05.086926 12721 tablet_service.cc:1514] Processing DeleteTablet for tablet cfcf39b1be5045ebb8beca0c6175a2b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:05 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:38874
I20260928 12:31:05.086954 12829 tablet_replica.cc:330] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: stopping tablet replica
I20260928 12:31:05.087179 12829 raft_consensus.cc:2229] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:05.087263 12830 tablet_replica.cc:330] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: stopping tablet replica
I20260928 12:31:05.087399 12829 raft_consensus.cc:2258] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:05.087543 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 12393
I20260928 12:31:05.088331 12830 raft_consensus.cc:2229] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:05.088510 12830 raft_consensus.cc:2258] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:05.089001 12830 ts_tablet_manager.cc:1905] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:05.089650 12829 ts_tablet_manager.cc:1905] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:05.090777 12830 ts_tablet_manager.cc:1918] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:05.090864 12830 log.cc:1196] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/cfcf39b1be5045ebb8beca0c6175a2b4
I20260928 12:31:05.091118 12829 ts_tablet_manager.cc:1918] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:05.091148 12830 ts_tablet_manager.cc:1939] T cfcf39b1be5045ebb8beca0c6175a2b4 P b5affc5b55fa46ffb39b152bb8443314: Deleting consensus metadata
I20260928 12:31:05.091194 12829 log.cc:1196] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/cfcf39b1be5045ebb8beca0c6175a2b4
I20260928 12:31:05.091406 12829 ts_tablet_manager.cc:1939] T cfcf39b1be5045ebb8beca0c6175a2b4 P 6466a4943b004b9eaf3a53c1af65fb34: Deleting consensus metadata
I20260928 12:31:05.091624 12316 catalog_manager.cc:4841] TS b5affc5b55fa46ffb39b152bb8443314 (127.8.13.3:43647): tablet cfcf39b1be5045ebb8beca0c6175a2b4 (table test-table [id=89d6b5327d8f457d8cfbb6d988092425]) successfully deleted
I20260928 12:31:05.091997 12315 catalog_manager.cc:4841] TS 6466a4943b004b9eaf3a53c1af65fb34 (127.8.13.2:40537): tablet cfcf39b1be5045ebb8beca0c6175a2b4 (table test-table [id=89d6b5327d8f457d8cfbb6d988092425]) successfully deleted
W20260928 12:31:05.093786 12318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:38647 (error 108)
I20260928 12:31:05.094115 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 12526
I20260928 12:31:05.094439 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.094303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:38647 (local address 127.0.0.1:38254)
0928 12:31:05.094369 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.094399 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 12:31:05.094751 12318 catalog_manager.cc:4584] TS ccefc8f914dc4cbfa6927e1311291419 (127.8.13.1:38647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfcf39b1be5045ebb8beca0c6175a2b4: Network error: Client connection negotiation failed: client connection to 127.8.13.1:38647: connect: Connection refused (error 111)
I20260928 12:31:05.099494 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 12659
I20260928 12:31:05.103808 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 12300
2026-09-28T12:31:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2166 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T12:31:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:05Z Disabled control of system clock
I20260928 12:31:05.131186 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33955
--webserver_interface=127.8.13.62
--webserver_port=0
--builtin_ntp_servers=127.8.13.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:33955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:31:05.214210 12835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:05.214430 12835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:05.214452 12835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:05.215704 12835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:05.215754 12835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:05.215766 12835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:05.215777 12835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:05.215788 12835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:05.217196 12835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:33955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:05.217434 12835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:05.217720 12835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:05.222548 12840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:05.222602 12843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:05.222762 12841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:05.222880 12835 server_base.cc:1034] running on GCE node
I20260928 12:31:05.223125 12835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:05.223408 12835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:05.224542 12835 hybrid_clock.cc:648] HybridClock initialized: now 1790598665224529 us; error 35 us; skew 500 ppm
I20260928 12:31:05.225667 12835 webserver.cc:460] Webserver started at http://127.8.13.62:41155/ using document root <none> and password file <none>
I20260928 12:31:05.225910 12835 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:05.225960 12835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:05.226092 12835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:05.227250 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "ef6f8bd543c34e909174162f0291231c"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "45474c5987d0303a6d686325b4fdaa30"
server_key_iv: "34c0b111e5453b95fd5984f88d0bbe7b"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.227608 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "ef6f8bd543c34e909174162f0291231c"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "45474c5987d0303a6d686325b4fdaa30"
server_key_iv: "34c0b111e5453b95fd5984f88d0bbe7b"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.228919 12835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:31:05.229674 12849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.229882 12835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:05.229966 12835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "ef6f8bd543c34e909174162f0291231c"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "45474c5987d0303a6d686325b4fdaa30"
server_key_iv: "34c0b111e5453b95fd5984f88d0bbe7b"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.230044 12835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:05.233398 12835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:05.233673 12835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:05.233834 12835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:05.238987 12835 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:33955
I20260928 12:31:05.239060 12901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:33955 every 8 connection(s)
I20260928 12:31:05.239408 12835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:05.240108 12902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:05.243187 12902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c: Bootstrap starting.
I20260928 12:31:05.244154 12902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:05.244536 12902 log.cc:824] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:05.245558 12902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c: No bootstrap required, opened a new log
I20260928 12:31:05.245597 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 12835
I20260928 12:31:05.245708 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:05.245941 8244 external_mini_cluster.cc:1428] Setting key 6f6d6673adfa1a104742490f9ed7801a
I20260928 12:31:05.248198 12902 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6f8bd543c34e909174162f0291231c" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33955 } }
I20260928 12:31:05.248392 12902 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:05.248426 12902 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6f8bd543c34e909174162f0291231c, State: Initialized, Role: FOLLOWER
I20260928 12:31:05.248601 12902 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6f8bd543c34e909174162f0291231c" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33955 } }
I20260928 12:31:05.248685 12902 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:05.248714 12902 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:05.248755 12902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:05.249532 12902 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6f8bd543c34e909174162f0291231c" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33955 } }
I20260928 12:31:05.249645 12902 leader_election.cc:302] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef6f8bd543c34e909174162f0291231c; no voters:
I20260928 12:31:05.249857 12902 leader_election.cc:288] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:05.249904 12907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:05.250008 12907 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [term 1 LEADER]: Becoming Leader. State: Replica: ef6f8bd543c34e909174162f0291231c, State: Running, Role: LEADER
I20260928 12:31:05.250097 12907 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6f8bd543c34e909174162f0291231c" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33955 } }
I20260928 12:31:05.250155 12902 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:05.250191 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.246646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.8.13.62:33955)
0928 12:31:05.246868 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.246877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.246905 (+ 28us) server_negotiation.cc:407] Connection header received
0928 12:31:05.248199 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.248212 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.248281 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.248490 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.249404 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.249409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.249409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.249415 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.249458 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.249470 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.249538 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.249610 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.249677 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260928 12:31:05.250737 12908 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef6f8bd543c34e909174162f0291231c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6f8bd543c34e909174162f0291231c" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33955 } } }
I20260928 12:31:05.250931 12908 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:05.251238 12909 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef6f8bd543c34e909174162f0291231c. Latest consensus state: current_term: 1 leader_uuid: "ef6f8bd543c34e909174162f0291231c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6f8bd543c34e909174162f0291231c" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33955 } } }
I20260928 12:31:05.251518 12920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:05.251524 12909 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:05.252024 12920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:05.253669 12920 catalog_manager.cc:1347] Generated new cluster ID: 045702a66cee463bb2f82ca77365f246
I20260928 12:31:05.253736 12920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:31:05.266350 12920 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:05.266583 12920 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:05.276177 12920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c: Generated new TSK 0
I20260928 12:31:05.276435 12920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:31:05.282056 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33955
--builtin_ntp_servers=127.8.13.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:31:05.365239 12926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:05.365494 12926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:05.365530 12926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:05.366940 12926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:05.367000 12926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:05.367043 12926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:05.368851 12926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:05.369158 12926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:05.369472 12926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:05.374712 12931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:05.374781 12932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:05.374716 12934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:05.375211 12926 server_base.cc:1034] running on GCE node
I20260928 12:31:05.375468 12926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:05.375774 12926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:05.376930 12926 hybrid_clock.cc:648] HybridClock initialized: now 1790598665376917 us; error 29 us; skew 500 ppm
I20260928 12:31:05.378629 12926 webserver.cc:460] Webserver started at http://127.8.13.1:35557/ using document root <none> and password file <none>
I20260928 12:31:05.378913 12926 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:05.378973 12926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:05.379101 12926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:05.380491 12926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "4da26fcba7c84c6b8f0aed4974ed873c"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "ff4860e6a332b579c43db3cf974de17b"
server_key_iv: "26cfaa19c6a29b25f171e35a3b16336d"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.380895 12926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "4da26fcba7c84c6b8f0aed4974ed873c"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "ff4860e6a332b579c43db3cf974de17b"
server_key_iv: "26cfaa19c6a29b25f171e35a3b16336d"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.382812 12926 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 12:31:05.383759 12940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.384018 12926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:05.384133 12926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "4da26fcba7c84c6b8f0aed4974ed873c"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "ff4860e6a332b579c43db3cf974de17b"
server_key_iv: "26cfaa19c6a29b25f171e35a3b16336d"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.384200 12926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:05.387715 12926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:05.387974 12926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:05.388075 12926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:05.388242 12926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:05.388506 12926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:05.388545 12926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.388568 12926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:05.388587 12926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.394184 12926 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:38775
I20260928 12:31:05.394232 13053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:38775 every 8 connection(s)
I20260928 12:31:05.394613 12926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:31:05.397464 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 12926
I20260928 12:31:05.397588 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:31:05.397787 8244 external_mini_cluster.cc:1428] Setting key d5624acc89189f53ee1799e5bd67cb51
I20260928 12:31:05.397996 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.395333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:33531 (local address 127.8.13.62:33955)
0928 12:31:05.395386 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.395388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.395616 (+ 228us) server_negotiation.cc:407] Connection header received
0928 12:31:05.396131 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.396133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.396141 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.396193 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.397597 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.397598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.397598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.397603 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.397621 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.397626 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.397681 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.397908 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.397926 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:31:05.398166 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.395314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33955 (local address 127.8.13.1:33531)
0928 12:31:05.395494 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.395529 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:05.396030 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:05.396267 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:05.396272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:05.396282 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:05.397424 (+ 1142us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:05.397425 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:05.397460 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:05.397497 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:05.397725 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:05.397734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:05.397738 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:05.397898 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 12:31:05.397977 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260928 12:31:05.398614 13054 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33955
I20260928 12:31:05.398725 13054 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:05.398839 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33955
--builtin_ntp_servers=127.8.13.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 12:31:05.398885 13054 heartbeater.cc:511] Master 127.8.13.62:33955 requested a full tablet report, sending...
I20260928 12:31:05.399401 12866 ts_manager.cc:194] Registered new tserver with Master: 4da26fcba7c84c6b8f0aed4974ed873c (127.8.13.1:38775)
W20260928 12:31:05.481678 13057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:05.481990 13057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:05.482023 13057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:05.483520 13057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:05.483577 13057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:05.483610 13057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:05.485152 13057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:05.485385 13057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:05.485672 13057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:05.490890 13065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:05.490880 13063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:05.490880 13062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:05.491173 13057 server_base.cc:1034] running on GCE node
I20260928 12:31:05.491393 13057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:05.491662 13057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:05.492841 13057 hybrid_clock.cc:648] HybridClock initialized: now 1790598665492826 us; error 35 us; skew 500 ppm
I20260928 12:31:05.494289 13057 webserver.cc:460] Webserver started at http://127.8.13.2:40139/ using document root <none> and password file <none>
I20260928 12:31:05.494537 13057 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:05.494588 13057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:05.494710 13057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:05.495775 13057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "114516acc651479f83ee8380f49d6633"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "5f84abd7fcbbe5265aff0021bc50ac7d"
server_key_iv: "c51840eb7f7065d84723820af62255e8"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.496101 13057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "114516acc651479f83ee8380f49d6633"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "5f84abd7fcbbe5265aff0021bc50ac7d"
server_key_iv: "c51840eb7f7065d84723820af62255e8"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.497454 13057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:31:05.498441 13071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.498711 13057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:05.498811 13057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "114516acc651479f83ee8380f49d6633"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "5f84abd7fcbbe5265aff0021bc50ac7d"
server_key_iv: "c51840eb7f7065d84723820af62255e8"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.498884 13057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:05.502319 13057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:05.502599 13057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:05.502722 13057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:05.502908 13057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:05.503198 13057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:05.503230 13057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.503257 13057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:05.503274 13057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.508735 13057 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:34365
I20260928 12:31:05.508785 13184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:34365 every 8 connection(s)
I20260928 12:31:05.509089 13057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:31:05.512017 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.509781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:43661 (local address 127.8.13.62:33955)
0928 12:31:05.509856 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.509858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.510088 (+ 230us) server_negotiation.cc:407] Connection header received
0928 12:31:05.510589 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.510591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.510597 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.510647 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.511772 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.511774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.511774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.511779 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.511797 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.511801 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.511860 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.511928 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.511943 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:31:05.512182 13187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.509778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33955 (local address 127.8.13.2:43661)
0928 12:31:05.509968 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.510002 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:05.510488 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:05.510716 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:05.510720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:05.510729 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:05.511623 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:05.511623 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:05.511651 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:05.511680 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:05.511855 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:05.511859 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:05.511862 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:05.511931 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 12:31:05.511993 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260928 12:31:05.512614 13185 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33955
I20260928 12:31:05.512706 13185 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:05.512844 13185 heartbeater.cc:511] Master 127.8.13.62:33955 requested a full tablet report, sending...
I20260928 12:31:05.513212 12866 ts_manager.cc:194] Registered new tserver with Master: 114516acc651479f83ee8380f49d6633 (127.8.13.2:34365)
I20260928 12:31:05.513584 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 13057
I20260928 12:31:05.513674 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:31:05.513878 8244 external_mini_cluster.cc:1428] Setting key 75ae81fdd691cf0c70d52a0b967a8657
I20260928 12:31:05.514683 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33955
--builtin_ntp_servers=127.8.13.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:31:05.597565 13188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:05.597790 13188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:05.597837 13188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:05.599135 13188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:05.599185 13188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:05.599216 13188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:05.600660 13188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:05.600886 13188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:05.601166 13188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:05.607317 13194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:05.607323 13193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:05.607303 13196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:05.607539 13188 server_base.cc:1034] running on GCE node
I20260928 12:31:05.607738 13188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:05.607997 13188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:05.609156 13188 hybrid_clock.cc:648] HybridClock initialized: now 1790598665609143 us; error 29 us; skew 500 ppm
I20260928 12:31:05.610467 13188 webserver.cc:460] Webserver started at http://127.8.13.3:44699/ using document root <none> and password file <none>
I20260928 12:31:05.610702 13188 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:05.610750 13188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:05.610872 13188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:05.611948 13188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "694033127431465cacfc16b7d3690912"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "1147a9725342264c2ba42f25257b396e"
server_key_iv: "619f7d39cff37ec9de3089abeedc6ff6"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.612269 13188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "694033127431465cacfc16b7d3690912"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "1147a9725342264c2ba42f25257b396e"
server_key_iv: "619f7d39cff37ec9de3089abeedc6ff6"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.613562 13188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:31:05.614359 13202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.614555 13188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.614646 13188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "694033127431465cacfc16b7d3690912"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "1147a9725342264c2ba42f25257b396e"
server_key_iv: "619f7d39cff37ec9de3089abeedc6ff6"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.614727 13188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:05.618017 13188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:05.618266 13188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:05.618383 13188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:05.618568 13188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:05.618841 13188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:05.618872 13188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.618898 13188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:05.618930 13188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.624548 13188 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:42769
I20260928 12:31:05.624595 13315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:42769 every 8 connection(s)
I20260928 12:31:05.624905 13188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:31:05.627893 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.625611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:48435 (local address 127.8.13.62:33955)
0928 12:31:05.625708 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.625710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.626001 (+ 291us) server_negotiation.cc:407] Connection header received
0928 12:31:05.626487 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.626488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.626495 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.626542 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.627661 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.627662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.627663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.627667 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.627681 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.627685 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.627735 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.627807 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.627822 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 12:31:05.628077 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.625593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33955 (local address 127.8.13.3:48435)
0928 12:31:05.625873 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.625914 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:05.626377 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:05.626623 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:05.626627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:05.626637 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:05.627516 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:05.627516 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:05.627545 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:05.627575 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:05.627737 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:05.627742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:05.627745 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:05.627812 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 12:31:05.627873 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260928 12:31:05.628496 13316 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33955
I20260928 12:31:05.628571 13316 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:05.628679 13316 heartbeater.cc:511] Master 127.8.13.62:33955 requested a full tablet report, sending...
I20260928 12:31:05.628979 12866 ts_manager.cc:194] Registered new tserver with Master: 694033127431465cacfc16b7d3690912 (127.8.13.3:42769)
I20260928 12:31:05.629418 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 13188
I20260928 12:31:05.629515 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:31:05.629681 8244 external_mini_cluster.cc:1428] Setting key 3b6d835879680c66018e050f0f511344
I20260928 12:31:05.630470 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:31:05.632582 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.631725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58160 (local address 127.8.13.62:33955)
0928 12:31:05.631765 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.631767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.631857 (+ 90us) server_negotiation.cc:407] Connection header received
0928 12:31:05.631908 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.631909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.631912 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.631962 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.632425 (+ 463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.632426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.632426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.632429 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.632442 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.632445 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.632482 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.632515 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.632525 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:31:05.633981 12866 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:31:05.635571 12866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:31:05.639529 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.638036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.8.13.2:34365)
0928 12:31:05.638380 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.638385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.638395 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:05.638418 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.638421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.638443 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.638784 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.639198 (+ 414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.639206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.639207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.639211 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.639248 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.639271 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.639335 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.639423 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.639440 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":81,"threads_started":1}
I20260928 12:31:05.639513 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.637929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:34365 (local address 127.0.0.1:33016)
0928 12:31:05.638156 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.638203 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:05.638269 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:05.638536 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:05.638541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:05.638548 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:05.639077 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:05.639078 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:05.639093 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:05.639119 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:05.639337 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:05.639343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:05.639347 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:05.639413 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 12:31:05.639428 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260928 12:31:05.639818 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.637978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:38775 (local address 127.0.0.1:39970)
0928 12:31:05.638193 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.638205 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:05.638269 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:05.639070 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:05.639071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:05.639075 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:05.639437 (+ 362us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:05.639438 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:05.639443 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:05.639454 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:05.639696 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:05.639697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:05.639698 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:31:05.639745 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:31:05.639758 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
I20260928 12:31:05.640125 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.638083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.8.13.1:38775)
0928 12:31:05.638274 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.638279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.638296 (+ 17us) server_negotiation.cc:407] Connection header received
0928 12:31:05.638384 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.638393 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.638407 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.638481 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.639527 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.639535 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.639535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.639540 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.639575 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.639599 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.639662 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.639761 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.639778 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20260928 12:31:05.640434 13119 tablet_service.cc:1467] Processing CreateTablet for tablet d72f27487f1a4e43aec2c65a909d0a7c (DEFAULT_TABLE table=test-table [id=c39095b749f44b82897628731e77ca9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:05.640676 12988 tablet_service.cc:1467] Processing CreateTablet for tablet d72f27487f1a4e43aec2c65a909d0a7c (DEFAULT_TABLE table=test-table [id=c39095b749f44b82897628731e77ca9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:05.640829 13119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72f27487f1a4e43aec2c65a909d0a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:05.640995 12988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72f27487f1a4e43aec2c65a909d0a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:05.644479 13330 tablet_bootstrap.cc:492] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633: Bootstrap starting.
I20260928 12:31:05.644752 13331 tablet_bootstrap.cc:492] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c: Bootstrap starting.
I20260928 12:31:05.645058 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.638176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:42769 (local address 127.0.0.1:54072)
0928 12:31:05.639356 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.639365 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:05.639550 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:05.640056 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:05.640057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:05.640060 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:05.640476 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:05.640476 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:05.640484 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:05.640496 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:05.644914 (+ 4418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:05.644915 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:05.644916 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:31:05.644966 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:31:05.644982 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"thread_start_us":75,"threads_started":1}
I20260928 12:31:05.645107 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.638369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.8.13.3:42769)
0928 12:31:05.638568 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.638572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.639530 (+ 958us) server_negotiation.cc:407] Connection header received
0928 12:31:05.639616 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.639620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.639856 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.639933 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.644772 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.644776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.644776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.644781 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.644810 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.644820 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.644884 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.645000 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.645023 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
I20260928 12:31:05.645725 13250 tablet_service.cc:1467] Processing CreateTablet for tablet d72f27487f1a4e43aec2c65a909d0a7c (DEFAULT_TABLE table=test-table [id=c39095b749f44b82897628731e77ca9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:05.645948 13330 tablet_bootstrap.cc:654] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:05.646096 13331 tablet_bootstrap.cc:654] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:05.646117 13250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72f27487f1a4e43aec2c65a909d0a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:05.646263 13330 log.cc:824] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:05.646379 13331 log.cc:824] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:05.647010 13330 tablet_bootstrap.cc:492] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633: No bootstrap required, opened a new log
I20260928 12:31:05.647063 13331 tablet_bootstrap.cc:492] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c: No bootstrap required, opened a new log
I20260928 12:31:05.647075 13330 ts_tablet_manager.cc:1397] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 12:31:05.647130 13331 ts_tablet_manager.cc:1397] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:31:05.648962 13330 raft_consensus.cc:348] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114516acc651479f83ee8380f49d6633" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34365 } } peers { permanent_uuid: "694033127431465cacfc16b7d3690912" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42769 } } peers { permanent_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38775 } }
I20260928 12:31:05.649111 13330 raft_consensus.cc:374] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:05.649143 13330 raft_consensus.cc:729] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 114516acc651479f83ee8380f49d6633, State: Initialized, Role: FOLLOWER
I20260928 12:31:05.649271 13330 consensus_queue.cc:260] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114516acc651479f83ee8380f49d6633" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34365 } } peers { permanent_uuid: "694033127431465cacfc16b7d3690912" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42769 } } peers { permanent_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38775 } }
I20260928 12:31:05.649569 13330 ts_tablet_manager.cc:1428] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 12:31:05.649572 13185 heartbeater.cc:503] Master 127.8.13.62:33955 was elected leader, sending a full tablet report...
I20260928 12:31:05.649977 13334 tablet_bootstrap.cc:492] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912: Bootstrap starting.
I20260928 12:31:05.650305 13331 raft_consensus.cc:348] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114516acc651479f83ee8380f49d6633" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34365 } } peers { permanent_uuid: "694033127431465cacfc16b7d3690912" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42769 } } peers { permanent_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38775 } }
I20260928 12:31:05.650478 13331 raft_consensus.cc:374] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:05.650514 13331 raft_consensus.cc:729] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da26fcba7c84c6b8f0aed4974ed873c, State: Initialized, Role: FOLLOWER
I20260928 12:31:05.650655 13331 consensus_queue.cc:260] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114516acc651479f83ee8380f49d6633" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34365 } } peers { permanent_uuid: "694033127431465cacfc16b7d3690912" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42769 } } peers { permanent_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38775 } }
I20260928 12:31:05.650712 13334 tablet_bootstrap.cc:654] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:05.650941 13331 ts_tablet_manager.cc:1428] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260928 12:31:05.650954 13054 heartbeater.cc:503] Master 127.8.13.62:33955 was elected leader, sending a full tablet report...
I20260928 12:31:05.651034 13334 log.cc:824] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:05.651822 13334 tablet_bootstrap.cc:492] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912: No bootstrap required, opened a new log
I20260928 12:31:05.651887 13334 ts_tablet_manager.cc:1397] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:31:05.653769 13334 raft_consensus.cc:348] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114516acc651479f83ee8380f49d6633" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34365 } } peers { permanent_uuid: "694033127431465cacfc16b7d3690912" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42769 } } peers { permanent_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38775 } }
I20260928 12:31:05.653982 13334 raft_consensus.cc:374] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:05.654027 13334 raft_consensus.cc:729] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694033127431465cacfc16b7d3690912, State: Initialized, Role: FOLLOWER
I20260928 12:31:05.654145 13334 consensus_queue.cc:260] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114516acc651479f83ee8380f49d6633" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34365 } } peers { permanent_uuid: "694033127431465cacfc16b7d3690912" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42769 } } peers { permanent_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38775 } }
I20260928 12:31:05.654389 13334 ts_tablet_manager.cc:1428] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 12:31:05.654433 13316 heartbeater.cc:503] Master 127.8.13.62:33955 was elected leader, sending a full tablet report...
I20260928 12:31:05.697688 13335 raft_consensus.cc:482] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:05.697840 13335 raft_consensus.cc:504] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114516acc651479f83ee8380f49d6633" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34365 } } peers { permanent_uuid: "694033127431465cacfc16b7d3690912" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42769 } } peers { permanent_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38775 } }
I20260928 12:31:05.698203 13335 leader_election.cc:288] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 694033127431465cacfc16b7d3690912 (127.8.13.3:42769), 4da26fcba7c84c6b8f0aed4974ed873c (127.8.13.1:38775)
I20260928 12:31:05.699623 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.698475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:40067 (local address 127.8.13.1:38775)
0928 12:31:05.698522 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.698524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.698588 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:31:05.698641 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.698642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.698649 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.698717 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.699321 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.699322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.699323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.699329 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.699348 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.699353 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.699419 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.699532 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.699549 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:31:05.699623 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.698478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:52651 (local address 127.8.13.3:42769)
0928 12:31:05.698532 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.698534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.698696 (+ 162us) server_negotiation.cc:407] Connection header received
0928 12:31:05.698726 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.698728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.698733 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.698785 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.699380 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.699382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.699382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.699387 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.699402 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.699407 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.699460 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.699535 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.699548 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:31:05.699631 13187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.698420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:38775 (local address 127.8.13.2:40067)
0928 12:31:05.698528 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.698539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:05.698588 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:05.698728 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:05.698729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:05.698732 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:05.699211 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:05.699211 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:05.699220 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:05.699239 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:05.699440 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:05.699442 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:05.699443 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:31:05.699526 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:31:05.699544 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 12:31:05.699644 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.698420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:42769 (local address 127.8.13.2:52651)
0928 12:31:05.698633 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.698641 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:05.698682 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:05.698794 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:05.698795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:05.698796 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:05.699277 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:05.699277 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:05.699282 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:05.699291 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:05.699478 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:05.699479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:05.699480 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:31:05.699532 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:31:05.699545 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":50,"thread_start_us":107,"threads_started":1}
I20260928 12:31:05.699978 13008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72f27487f1a4e43aec2c65a909d0a7c" candidate_uuid: "114516acc651479f83ee8380f49d6633" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" is_pre_election: true
I20260928 12:31:05.700193 13008 raft_consensus.cc:2454] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 114516acc651479f83ee8380f49d6633 in term 0.
I20260928 12:31:05.700487 13073 leader_election.cc:302] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114516acc651479f83ee8380f49d6633, 4da26fcba7c84c6b8f0aed4974ed873c; no voters:
I20260928 12:31:05.700501 13270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72f27487f1a4e43aec2c65a909d0a7c" candidate_uuid: "114516acc651479f83ee8380f49d6633" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694033127431465cacfc16b7d3690912" is_pre_election: true
I20260928 12:31:05.700608 13335 raft_consensus.cc:2781] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:31:05.700637 13270 raft_consensus.cc:2454] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 114516acc651479f83ee8380f49d6633 in term 0.
I20260928 12:31:05.700659 13335 raft_consensus.cc:482] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:31:05.700681 13335 raft_consensus.cc:3037] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:05.701427 13335 raft_consensus.cc:504] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114516acc651479f83ee8380f49d6633" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34365 } } peers { permanent_uuid: "694033127431465cacfc16b7d3690912" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42769 } } peers { permanent_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38775 } }
I20260928 12:31:05.701565 13335 leader_election.cc:288] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [CANDIDATE]: Term 1 election: Requested vote from peers 694033127431465cacfc16b7d3690912 (127.8.13.3:42769), 4da26fcba7c84c6b8f0aed4974ed873c (127.8.13.1:38775)
I20260928 12:31:05.701735 13008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72f27487f1a4e43aec2c65a909d0a7c" candidate_uuid: "114516acc651479f83ee8380f49d6633" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da26fcba7c84c6b8f0aed4974ed873c"
I20260928 12:31:05.701774 13270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72f27487f1a4e43aec2c65a909d0a7c" candidate_uuid: "114516acc651479f83ee8380f49d6633" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694033127431465cacfc16b7d3690912"
I20260928 12:31:05.701896 13270 raft_consensus.cc:3037] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:05.701898 13008 raft_consensus.cc:3037] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:05.702621 13270 raft_consensus.cc:2454] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 114516acc651479f83ee8380f49d6633 in term 1.
I20260928 12:31:05.702993 13008 raft_consensus.cc:2454] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 114516acc651479f83ee8380f49d6633 in term 1.
I20260928 12:31:05.703177 13073 leader_election.cc:302] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114516acc651479f83ee8380f49d6633, 4da26fcba7c84c6b8f0aed4974ed873c; no voters:
I20260928 12:31:05.703293 13335 raft_consensus.cc:2781] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:05.703460 13335 raft_consensus.cc:686] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 1 LEADER]: Becoming Leader. State: Replica: 114516acc651479f83ee8380f49d6633, State: Running, Role: LEADER
I20260928 12:31:05.703565 13335 consensus_queue.cc:237] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114516acc651479f83ee8380f49d6633" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34365 } } peers { permanent_uuid: "694033127431465cacfc16b7d3690912" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42769 } } peers { permanent_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38775 } }
I20260928 12:31:05.704493 12866 catalog_manager.cc:5495] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 reported cstate change: term changed from 0 to 1, leader changed from <none> to 114516acc651479f83ee8380f49d6633 (127.8.13.2). New cstate: current_term: 1 leader_uuid: "114516acc651479f83ee8380f49d6633" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114516acc651479f83ee8380f49d6633" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "694033127431465cacfc16b7d3690912" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38775 } health_report { overall_health: UNKNOWN } } }
I20260928 12:31:05.714787 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.713893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.8.13.2:34365)
0928 12:31:05.713959 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.713961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.713973 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:31:05.714017 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.714018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.714030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.714078 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.714571 (+ 493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.714572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.714573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.714577 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.714592 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.714596 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.714644 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.714696 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.714709 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:31:05.716521 13270 raft_consensus.cc:1261] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912 [term 1 FOLLOWER]: Refusing update from remote peer 114516acc651479f83ee8380f49d6633: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:05.716519 13008 raft_consensus.cc:1261] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c [term 1 FOLLOWER]: Refusing update from remote peer 114516acc651479f83ee8380f49d6633: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:05.716900 13340 consensus_queue.cc:1035] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [LEADER]: Connected to new peer: Peer: permanent_uuid: "694033127431465cacfc16b7d3690912" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:05.717032 13335 consensus_queue.cc:1035] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da26fcba7c84c6b8f0aed4974ed873c" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 38775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:05.719851 13344 mvcc.cc:205] Tried to move back new op lower bound from 7334292134772326400 to 7334292134722469888. Current Snapshot: MvccSnapshot[applied={T|T < 7334292134772326400}]
I20260928 12:31:05.721575 12864 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:05.721659 12864 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:05.722386 12864 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef6f8bd543c34e909174162f0291231c: Sending DeleteTablet for 3 replicas of tablet d72f27487f1a4e43aec2c65a909d0a7c
I20260928 12:31:05.722683 13119 tablet_service.cc:1514] Processing DeleteTablet for tablet d72f27487f1a4e43aec2c65a909d0a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:05 UTC) from {username='slave'} at 127.0.0.1:33016
I20260928 12:31:05.722887 13353 tablet_replica.cc:330] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633: stopping tablet replica
I20260928 12:31:05.722887 12988 tablet_service.cc:1514] Processing DeleteTablet for tablet d72f27487f1a4e43aec2c65a909d0a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:05 UTC) from {username='slave'} at 127.0.0.1:39970
I20260928 12:31:05.723022 13353 raft_consensus.cc:2229] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:31:05.723150 13353 raft_consensus.cc:2258] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:05.723132 13250 tablet_service.cc:1514] Processing DeleteTablet for tablet d72f27487f1a4e43aec2c65a909d0a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:05 UTC) from {username='slave'} at 127.0.0.1:54072
I20260928 12:31:05.723342 13355 tablet_replica.cc:330] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912: stopping tablet replica
I20260928 12:31:05.723404 13354 tablet_replica.cc:330] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c: stopping tablet replica
I20260928 12:31:05.723491 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 12926
I20260928 12:31:05.723541 13354 raft_consensus.cc:2229] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:05.723564 13355 raft_consensus.cc:2229] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:05.723641 13354 raft_consensus.cc:2258] T d72f27487f1a4e43aec2c65a909d0a7c P 4da26fcba7c84c6b8f0aed4974ed873c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:05.723989 13355 raft_consensus.cc:2258] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:05.724486 13355 ts_tablet_manager.cc:1905] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:05.726468 13355 ts_tablet_manager.cc:1918] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:05.726554 13355 log.cc:1196] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/d72f27487f1a4e43aec2c65a909d0a7c
I20260928 12:31:05.726827 13355 ts_tablet_manager.cc:1939] T d72f27487f1a4e43aec2c65a909d0a7c P 694033127431465cacfc16b7d3690912: Deleting consensus metadata
I20260928 12:31:05.727249 12852 catalog_manager.cc:4841] TS 694033127431465cacfc16b7d3690912 (127.8.13.3:42769): tablet d72f27487f1a4e43aec2c65a909d0a7c (table test-table [id=c39095b749f44b82897628731e77ca9c]) successfully deleted
I20260928 12:31:05.727445 13353 ts_tablet_manager.cc:1905] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:05.729014 13353 ts_tablet_manager.cc:1918] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:05.729096 13353 log.cc:1196] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/d72f27487f1a4e43aec2c65a909d0a7c
W20260928 12:31:05.729206 12851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:38775 (error 108)
I20260928 12:31:05.729338 13353 ts_tablet_manager.cc:1939] T d72f27487f1a4e43aec2c65a909d0a7c P 114516acc651479f83ee8380f49d6633: Deleting consensus metadata
I20260928 12:31:05.729700 12852 catalog_manager.cc:4841] TS 114516acc651479f83ee8380f49d6633 (127.8.13.2:34365): tablet d72f27487f1a4e43aec2c65a909d0a7c (table test-table [id=c39095b749f44b82897628731e77ca9c]) successfully deleted
I20260928 12:31:05.729851 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.729603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:38775 (local address 127.0.0.1:39976)
0928 12:31:05.729648 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.729685 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:31:05.729952 12851 catalog_manager.cc:4584] TS 4da26fcba7c84c6b8f0aed4974ed873c (127.8.13.1:38775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d72f27487f1a4e43aec2c65a909d0a7c: Network error: Client connection negotiation failed: client connection to 127.8.13.1:38775: connect: Connection refused (error 111)
I20260928 12:31:05.730181 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 13057
I20260928 12:31:05.734408 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 13188
I20260928 12:31:05.738746 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 12835
2026-09-28T12:31:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (633 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 12:31:05.758100 8244 mini_oidc.cc:152] Starting JWKS server
I20260928 12:31:05.759438 8244 webserver.cc:460] Webserver started at https://127.0.0.1:33723/ using document root <none> and password file <none>
I20260928 12:31:05.759559 8244 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:31:05.759743 8244 webserver.cc:460] Webserver started at http://127.0.0.1:34551/ using document root <none> and password file <none>
2026-09-28T12:31:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:05Z Disabled control of system clock
I20260928 12:31:05.768464 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:43107
--webserver_interface=127.8.13.62
--webserver_port=0
--builtin_ntp_servers=127.8.13.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:43107
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33723/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/testchainca.cert with env {}
W20260928 12:31:05.849864 13362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:05.850090 13362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:05.850113 13362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:05.852067 13362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:05.852180 13362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:05.852195 13362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:05.852208 13362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:05.852219 13362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:31:05.852233 13362 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:31:05.852243 13362 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33723/jwks.json
I20260928 12:31:05.855443 13362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:43107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:43107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33723/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:05.855883 13362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:05.856140 13362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:05.861230 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:05.861245 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:05.861320 13362 server_base.cc:1034] running on GCE node
W20260928 12:31:05.861250 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:05.861690 13362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:05.862013 13362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:05.863169 13362 hybrid_clock.cc:648] HybridClock initialized: now 1790598665863152 us; error 33 us; skew 500 ppm
I20260928 12:31:05.864476 13362 webserver.cc:460] Webserver started at http://127.8.13.62:46179/ using document root <none> and password file <none>
I20260928 12:31:05.864717 13362 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:05.864765 13362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:05.864876 13362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:05.866012 13362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "1245d0537d324d5da8b3551ee1ba8d54"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "0e3157e433e51181b32301e34141b6ee"
server_key_iv: "23334c228755dcd948fdba6442e16758"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.866379 13362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "1245d0537d324d5da8b3551ee1ba8d54"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "0e3157e433e51181b32301e34141b6ee"
server_key_iv: "23334c228755dcd948fdba6442e16758"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.868007 13362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:05.868830 13376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:05.869122 13362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:05.869297 13362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "1245d0537d324d5da8b3551ee1ba8d54"
format_stamp: "Formatted at 2026-09-28 12:31:05 on dist-test-slave-g31s"
server_key: "0e3157e433e51181b32301e34141b6ee"
server_key_iv: "23334c228755dcd948fdba6442e16758"
server_key_version: "encryptionkey@0"
I20260928 12:31:05.869414 13362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:05.880272 13362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:05.880622 13362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:05.880741 13362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:05.884790 13362 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:43107
I20260928 12:31:05.884840 13428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:43107 every 8 connection(s)
I20260928 12:31:05.885160 13362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:05.886369 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:05.889189 13429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54: Bootstrap starting.
I20260928 12:31:05.889940 13429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:05.890260 13429 log.cc:824] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:05.891346 13429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54: No bootstrap required, opened a new log
I20260928 12:31:05.893740 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 13362
I20260928 12:31:05.893649 13429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1245d0537d324d5da8b3551ee1ba8d54" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43107 } }
I20260928 12:31:05.893896 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:05.893906 13429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:05.893946 13429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1245d0537d324d5da8b3551ee1ba8d54, State: Initialized, Role: FOLLOWER
I20260928 12:31:05.894095 8244 external_mini_cluster.cc:1428] Setting key 241b7dce19cf3bab99092bc96b6b9cc4
I20260928 12:31:05.894122 13429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1245d0537d324d5da8b3551ee1ba8d54" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43107 } }
I20260928 12:31:05.894212 13429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:05.894254 13429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:05.894304 13429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:05.895131 13429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1245d0537d324d5da8b3551ee1ba8d54" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43107 } }
I20260928 12:31:05.895251 13429 leader_election.cc:302] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1245d0537d324d5da8b3551ee1ba8d54; no voters:
I20260928 12:31:05.895424 13429 leader_election.cc:288] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:05.895483 13434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:05.895745 13429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:05.895742 13434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [term 1 LEADER]: Becoming Leader. State: Replica: 1245d0537d324d5da8b3551ee1ba8d54, State: Running, Role: LEADER
I20260928 12:31:05.895952 13434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1245d0537d324d5da8b3551ee1ba8d54" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43107 } }
I20260928 12:31:05.896347 13436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1245d0537d324d5da8b3551ee1ba8d54. Latest consensus state: current_term: 1 leader_uuid: "1245d0537d324d5da8b3551ee1ba8d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1245d0537d324d5da8b3551ee1ba8d54" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43107 } } }
I20260928 12:31:05.896445 13436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:05.896646 13443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:05.897137 13443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:05.898180 13435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1245d0537d324d5da8b3551ee1ba8d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1245d0537d324d5da8b3551ee1ba8d54" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43107 } } }
I20260928 12:31:05.898263 13435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:05.899407 13443 catalog_manager.cc:1347] Generated new cluster ID: 4503a949a32846d4a8b3546f9439d137
I20260928 12:31:05.899453 13443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:31:05.900635 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.894614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39548 (local address 127.8.13.62:43107)
0928 12:31:05.894951 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.894961 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.895006 (+ 45us) server_negotiation.cc:407] Connection header received
0928 12:31:05.895146 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.895160 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.895240 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.895458 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:05.895864 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.897547 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.898494 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.899071 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.899193 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:05.899911 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:05.899915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:05.899916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:05.900130 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:05.900172 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:05.900187 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:05.900250 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:05.900334 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 12:31:05.900424 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
I20260928 12:31:05.901612 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:31:05.918725 13443 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:05.919502 13443 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:05.927042 13443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1245d0537d324d5da8b3551ee1ba8d54: Generated new TSK 0
I20260928 12:31:05.927253 13443 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 12:31:05.957464 13459 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 12:31:05.957587 13459 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 12:31:05.957826 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:05.942510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.8.13.62:43107)
0928 12:31:05.942563 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:05.942565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:05.942686 (+ 121us) server_negotiation.cc:407] Connection header received
0928 12:31:05.942807 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:05.942808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:05.942818 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:05.942877 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:31:05.943061 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.943554 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.944743 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.944957 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.945020 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:05.945277 (+ 257us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:31:05.957559 (+ 12282us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 12:31:05.957733 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39558: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 12:31:05.957880 13433 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39558: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 12:31:05.958204 13458 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:05.942378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:43107 (local address 127.0.0.1:39558)
0928 12:31:05.942604 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:05.942617 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:05.942707 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:05.942896 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:05.942897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:05.942899 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:31:05.943003 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:05.943006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.943643 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.943644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:05.944634 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:05.944636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:05.945028 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:05.945033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:05.945097 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:05.945127 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:31:05.957782 (+ 12655us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 12:31:05.957940 (+ 158us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.13.62:43107: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":118,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.13.62:43107: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 12:31:05.958595 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 13362
2026-09-28T12:31:05Z chronyd exiting
I20260928 12:31:05.994828 8244 test_util.cc:175] -----------------------------------------------
I20260928 12:31:05.994936 8244 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598640018661-8244-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 12:31:05.995797 8244 mini_oidc.cc:152] Starting JWKS server
I20260928 12:31:05.996512 8244 webserver.cc:460] Webserver started at https://127.0.0.1:39647/ using document root <none> and password file <none>
I20260928 12:31:05.996685 8244 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:31:05.996989 8244 webserver.cc:460] Webserver started at http://127.0.0.1:44705/ using document root <none> and password file <none>
2026-09-28T12:31:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:05Z Disabled control of system clock
I20260928 12:31:06.004556 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:43933
--webserver_interface=127.8.13.62
--webserver_port=0
--builtin_ntp_servers=127.8.13.20:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:43933
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39647/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/testchainca.cert with env {}
W20260928 12:31:06.084568 13465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:06.084818 13465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:06.084852 13465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:06.086205 13465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:06.086262 13465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:06.086280 13465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:06.086297 13465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:06.086313 13465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:31:06.086337 13465 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:31:06.086361 13465 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39647/jwks.json
I20260928 12:31:06.088109 13465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:43933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:43933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39647/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:06.088435 13465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:06.088651 13465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:06.093570 13471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:06.093565 13470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:06.093611 13465 server_base.cc:1034] running on GCE node
W20260928 12:31:06.093617 13473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:06.094223 13465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:06.094581 13465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:06.095758 13465 hybrid_clock.cc:648] HybridClock initialized: now 1790598666095724 us; error 70 us; skew 500 ppm
I20260928 12:31:06.097213 13465 webserver.cc:460] Webserver started at http://127.8.13.62:38267/ using document root <none> and password file <none>
I20260928 12:31:06.097457 13465 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:06.097512 13465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:06.097661 13465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:06.098945 13465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "a821453f93ec4023be49bd4fd6f9c1c3"
format_stamp: "Formatted at 2026-09-28 12:31:06 on dist-test-slave-g31s"
server_key: "ac68ac1e9dc2119e09d7c4fd6b85817b"
server_key_iv: "6e374bf5b0039c2edd9423bb39a00b8a"
server_key_version: "encryptionkey@0"
I20260928 12:31:06.099568 13465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "a821453f93ec4023be49bd4fd6f9c1c3"
format_stamp: "Formatted at 2026-09-28 12:31:06 on dist-test-slave-g31s"
server_key: "ac68ac1e9dc2119e09d7c4fd6b85817b"
server_key_iv: "6e374bf5b0039c2edd9423bb39a00b8a"
server_key_version: "encryptionkey@0"
I20260928 12:31:06.101137 13465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:31:06.101981 13479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:06.102200 13465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:06.102295 13465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "a821453f93ec4023be49bd4fd6f9c1c3"
format_stamp: "Formatted at 2026-09-28 12:31:06 on dist-test-slave-g31s"
server_key: "ac68ac1e9dc2119e09d7c4fd6b85817b"
server_key_iv: "6e374bf5b0039c2edd9423bb39a00b8a"
server_key_version: "encryptionkey@0"
I20260928 12:31:06.102365 13465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:06.122761 13465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:06.123088 13465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:06.123219 13465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:06.127246 13465 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:43933
I20260928 12:31:06.127290 13531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:43933 every 8 connection(s)
I20260928 12:31:06.127696 13465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:06.128916 13532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:06.129235 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 13465
I20260928 12:31:06.129334 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:06.129494 8244 external_mini_cluster.cc:1428] Setting key 86428634b7e83bb423fdeed741afab51
I20260928 12:31:06.132038 13532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3: Bootstrap starting.
I20260928 12:31:06.132740 13532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:06.133000 13532 log.cc:824] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:06.133843 13532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3: No bootstrap required, opened a new log
I20260928 12:31:06.135737 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:06.130016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55804 (local address 127.8.13.62:43933)
0928 12:31:06.130247 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:06.130256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:06.130306 (+ 50us) server_negotiation.cc:407] Connection header received
0928 12:31:06.130815 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:06.130823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:06.130905 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:06.131059 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:06.131258 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.132539 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.133483 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.134152 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.134283 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:06.134937 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:06.134942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:06.134942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:06.135154 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:06.135198 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:06.135214 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:06.135293 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:06.135408 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 12:31:06.135504 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260928 12:31:06.136360 13532 raft_consensus.cc:348] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821453f93ec4023be49bd4fd6f9c1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43933 } }
I20260928 12:31:06.136559 13532 raft_consensus.cc:374] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:06.136590 13532 raft_consensus.cc:729] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a821453f93ec4023be49bd4fd6f9c1c3, State: Initialized, Role: FOLLOWER
I20260928 12:31:06.136694 13532 consensus_queue.cc:260] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821453f93ec4023be49bd4fd6f9c1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43933 } }
I20260928 12:31:06.136770 13532 raft_consensus.cc:388] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:06.136790 13532 raft_consensus.cc:482] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:06.136835 13532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:06.137120 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:31:06.137550 13532 raft_consensus.cc:504] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821453f93ec4023be49bd4fd6f9c1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43933 } }
I20260928 12:31:06.137653 13532 leader_election.cc:302] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a821453f93ec4023be49bd4fd6f9c1c3; no voters:
I20260928 12:31:06.137851 13532 leader_election.cc:288] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:06.137895 13537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:06.138058 13537 raft_consensus.cc:686] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [term 1 LEADER]: Becoming Leader. State: Replica: a821453f93ec4023be49bd4fd6f9c1c3, State: Running, Role: LEADER
I20260928 12:31:06.138120 13532 sys_catalog.cc:564] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:06.138183 13537 consensus_queue.cc:237] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821453f93ec4023be49bd4fd6f9c1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43933 } }
I20260928 12:31:06.138516 13537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a821453f93ec4023be49bd4fd6f9c1c3. Latest consensus state: current_term: 1 leader_uuid: "a821453f93ec4023be49bd4fd6f9c1c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821453f93ec4023be49bd4fd6f9c1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43933 } } }
I20260928 12:31:06.138540 13539 sys_catalog.cc:455] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a821453f93ec4023be49bd4fd6f9c1c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821453f93ec4023be49bd4fd6f9c1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43933 } } }
I20260928 12:31:06.138590 13537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:06.138621 13539 sys_catalog.cc:458] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:06.138784 13544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:06.139307 13544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:06.140997 13544 catalog_manager.cc:1347] Generated new cluster ID: 9fe7ebd9b78f482db69ae498f0ec5f9a
I20260928 12:31:06.141064 13544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:31:06.149290 13544 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:06.150084 13544 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:06.156005 13544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a821453f93ec4023be49bd4fd6f9c1c3: Generated new TSK 0
I20260928 12:31:06.156214 13544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:31:06.185091 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:06.168172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.8.13.62:43933)
0928 12:31:06.168221 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:06.168223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:06.168284 (+ 61us) server_negotiation.cc:407] Connection header received
0928 12:31:06.168386 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:06.168386 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:06.168399 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:06.168463 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:31:06.168602 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.169115 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.170341 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.170564 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.170631 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:06.170800 (+ 169us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:31:06.184805 (+ 14005us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 12:31:06.185000 (+ 195us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 12:31:06.185233 13561 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:06.168078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:43933 (local address 127.0.0.1:55818)
0928 12:31:06.168221 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:06.168233 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:06.168312 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:06.168475 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:06.168477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:06.168478 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:31:06.168542 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:06.168545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.169197 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.169198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:06.170206 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:06.170211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.170635 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.170640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:06.170703 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:06.170709 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:31:06.184966 (+ 14257us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 12:31:06.185117 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.13.62:43933: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260928 12:31:06.185252 13535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
E20260928 12:31:06.185544 13562 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 12:31:06.185617 13562 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 12:31:06.186260 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 13465
2026-09-28T12:31:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 12:31:06.223348 8244 mini_oidc.cc:152] Starting JWKS server
I20260928 12:31:06.224020 8244 webserver.cc:460] Webserver started at https://127.0.0.1:43827/ using document root <none> and password file <none>
I20260928 12:31:06.224063 8244 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:31:06.224169 8244 webserver.cc:460] Webserver started at http://127.0.0.1:45825/ using document root <none> and password file <none>
2026-09-28T12:31:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:06Z Disabled control of system clock
I20260928 12:31:06.231310 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:37147
--webserver_interface=127.8.13.62
--webserver_port=0
--builtin_ntp_servers=127.8.13.20:46449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:37147
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43827/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 12:31:06.317358 13568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:06.317605 13568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:06.317627 13568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:06.318857 13568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:06.318903 13568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:06.318915 13568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:06.318926 13568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:06.318938 13568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:31:06.318949 13568 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:31:06.318961 13568 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43827/jwks.json
I20260928 12:31:06.320345 13568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:46449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:37147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:37147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43827/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:06.320572 13568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:06.320844 13568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:06.325591 13576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:06.325676 13573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:06.325595 13574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:06.325875 13568 server_base.cc:1034] running on GCE node
I20260928 12:31:06.326108 13568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:06.326407 13568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:06.327569 13568 hybrid_clock.cc:648] HybridClock initialized: now 1790598666327549 us; error 41 us; skew 500 ppm
I20260928 12:31:06.328972 13568 webserver.cc:460] Webserver started at http://127.8.13.62:40701/ using document root <none> and password file <none>
I20260928 12:31:06.329206 13568 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:06.329258 13568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:06.329391 13568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:06.330557 13568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "b92cb40c58f04985bd09d1597c6ee044"
format_stamp: "Formatted at 2026-09-28 12:31:06 on dist-test-slave-g31s"
server_key: "16aabcabbb2a5d6a30b019985eabc758"
server_key_iv: "287bf72e477145ebcefc0048cdb74f95"
server_key_version: "encryptionkey@0"
I20260928 12:31:06.330914 13568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "b92cb40c58f04985bd09d1597c6ee044"
format_stamp: "Formatted at 2026-09-28 12:31:06 on dist-test-slave-g31s"
server_key: "16aabcabbb2a5d6a30b019985eabc758"
server_key_iv: "287bf72e477145ebcefc0048cdb74f95"
server_key_version: "encryptionkey@0"
I20260928 12:31:06.332254 13568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:06.333060 13582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:06.333316 13568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:06.333472 13568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "b92cb40c58f04985bd09d1597c6ee044"
format_stamp: "Formatted at 2026-09-28 12:31:06 on dist-test-slave-g31s"
server_key: "16aabcabbb2a5d6a30b019985eabc758"
server_key_iv: "287bf72e477145ebcefc0048cdb74f95"
server_key_version: "encryptionkey@0"
I20260928 12:31:06.333570 13568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:06.347353 13568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:06.347754 13568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:06.347900 13568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:06.351955 13568 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:37147
I20260928 12:31:06.351999 13634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:37147 every 8 connection(s)
I20260928 12:31:06.352349 13568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:06.353197 13635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:06.355835 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 13568
I20260928 12:31:06.355967 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:06.356142 8244 external_mini_cluster.cc:1428] Setting key 3c809681910077401a9a33b27481ed72
I20260928 12:31:06.356180 13635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044: Bootstrap starting.
I20260928 12:31:06.356966 13635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:06.357328 13635 log.cc:824] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:06.358233 13635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044: No bootstrap required, opened a new log
I20260928 12:31:06.360565 13635 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92cb40c58f04985bd09d1597c6ee044" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37147 } }
I20260928 12:31:06.360791 13635 raft_consensus.cc:374] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:06.360829 13635 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92cb40c58f04985bd09d1597c6ee044, State: Initialized, Role: FOLLOWER
I20260928 12:31:06.360984 13635 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92cb40c58f04985bd09d1597c6ee044" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37147 } }
I20260928 12:31:06.361066 13635 raft_consensus.cc:388] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:06.361099 13635 raft_consensus.cc:482] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:06.361147 13635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:06.362051 13635 raft_consensus.cc:504] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92cb40c58f04985bd09d1597c6ee044" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37147 } }
I20260928 12:31:06.362192 13635 leader_election.cc:302] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b92cb40c58f04985bd09d1597c6ee044; no voters:
I20260928 12:31:06.362421 13635 leader_election.cc:288] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:06.362469 13640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:06.362627 13640 raft_consensus.cc:686] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [term 1 LEADER]: Becoming Leader. State: Replica: b92cb40c58f04985bd09d1597c6ee044, State: Running, Role: LEADER
I20260928 12:31:06.362721 13640 consensus_queue.cc:237] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92cb40c58f04985bd09d1597c6ee044" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37147 } }
I20260928 12:31:06.362739 13635 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:06.363154 13641 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b92cb40c58f04985bd09d1597c6ee044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92cb40c58f04985bd09d1597c6ee044" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37147 } } }
I20260928 12:31:06.363271 13641 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:06.363149 13642 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92cb40c58f04985bd09d1597c6ee044. Latest consensus state: current_term: 1 leader_uuid: "b92cb40c58f04985bd09d1597c6ee044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92cb40c58f04985bd09d1597c6ee044" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37147 } } }
I20260928 12:31:06.363437 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:06.356821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.8.13.62:37147)
0928 12:31:06.357153 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:06.357166 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:06.357207 (+ 41us) server_negotiation.cc:407] Connection header received
0928 12:31:06.357764 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:06.357773 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:06.357886 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:06.358017 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:06.358239 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.359370 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.360646 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.361214 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.361354 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:06.362370 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:06.362375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:06.362376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:06.362621 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:06.362674 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:06.362691 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:06.363024 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:06.363061 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 12:31:06.363152 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":149,"threads_started":1}
I20260928 12:31:06.363452 13642 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:06.364364 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 12:31:06.364759 13655 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:31:06.364820 13655 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:31:06.364940 13647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:06.365114 13647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:06.366998 13647 catalog_manager.cc:1347] Generated new cluster ID: 515062fc5cd3482484e589955c50e26b
I20260928 12:31:06.367125 13647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:31:06.383343 13647 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:06.384101 13647 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:06.395965 13647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b92cb40c58f04985bd09d1597c6ee044: Generated new TSK 0
I20260928 12:31:06.396204 13647 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:31:06.413910 13664 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:06.410088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37147 (local address 127.0.0.1:48072)
0928 12:31:06.410255 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:06.410266 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:06.410347 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:06.410526 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:06.410526 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:06.410528 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:31:06.410589 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:06.410592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.411224 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.411224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:06.412174 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:06.412177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.412526 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.412534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:06.412587 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:06.412593 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:31:06.413677 (+ 1084us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 12:31:06.413783 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.13.62:37147: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260928 12:31:06.413988 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:06.410155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.8.13.62:37147)
0928 12:31:06.410215 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:06.410217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:06.410335 (+ 118us) server_negotiation.cc:407] Connection header received
0928 12:31:06.410426 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:06.410427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:06.410455 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:06.410517 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:31:06.410645 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.411160 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.412254 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.412463 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.412516 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:06.412681 (+ 165us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:31:06.413576 (+ 895us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 12:31:06.413708 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48072: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 12:31:06.414042 13638 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48072: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260928 12:31:06.414156 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 13568
E20260928 12:31:06.424746 13665 webserver.cc:533] Webserver: sslize failed:
E20260928 12:31:06.424836 13665 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T12:31:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T12:31:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:06Z Disabled control of system clock
I20260928 12:31:06.458797 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:42273
--webserver_interface=127.8.13.62
--webserver_port=0
--builtin_ntp_servers=127.8.13.20:38855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:42273
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/testchainca.cert with env {}
W20260928 12:31:06.544777 13669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:06.545022 13669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:06.545048 13669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:06.546352 13669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:06.546406 13669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:06.546417 13669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:06.546428 13669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:06.546439 13669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:31:06.546451 13669 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:31:06.546461 13669 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 12:31:06.548138 13669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:38855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:42273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:42273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:06.548400 13669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:06.548580 13669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:06.553645 13677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:06.554004 13669 server_base.cc:1034] running on GCE node
W20260928 12:31:06.553675 13675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:06.553857 13674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:06.554512 13669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:06.554839 13669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:06.556011 13669 hybrid_clock.cc:648] HybridClock initialized: now 1790598666555959 us; error 68 us; skew 500 ppm
I20260928 12:31:06.557426 13669 webserver.cc:460] Webserver started at http://127.8.13.62:43471/ using document root <none> and password file <none>
I20260928 12:31:06.557662 13669 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:06.557714 13669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:06.557881 13669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:06.559019 13669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "35c6d0624cb44ab291d96c7e64768ee6"
format_stamp: "Formatted at 2026-09-28 12:31:06 on dist-test-slave-g31s"
server_key: "86be4b737e7bdc7739ac9ea16a00c2bf"
server_key_iv: "516e4caa763af3cbe60b4c0627fe7087"
server_key_version: "encryptionkey@0"
I20260928 12:31:06.559372 13669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "35c6d0624cb44ab291d96c7e64768ee6"
format_stamp: "Formatted at 2026-09-28 12:31:06 on dist-test-slave-g31s"
server_key: "86be4b737e7bdc7739ac9ea16a00c2bf"
server_key_iv: "516e4caa763af3cbe60b4c0627fe7087"
server_key_version: "encryptionkey@0"
I20260928 12:31:06.560832 13669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:31:06.562023 13683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:06.562305 13669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:06.562422 13669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "35c6d0624cb44ab291d96c7e64768ee6"
format_stamp: "Formatted at 2026-09-28 12:31:06 on dist-test-slave-g31s"
server_key: "86be4b737e7bdc7739ac9ea16a00c2bf"
server_key_iv: "516e4caa763af3cbe60b4c0627fe7087"
server_key_version: "encryptionkey@0"
I20260928 12:31:06.562511 13669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:06.584187 13669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:06.584579 13669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:06.584720 13669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:06.588663 13669 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:42273
I20260928 12:31:06.588711 13735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:42273 every 8 connection(s)
I20260928 12:31:06.589041 13669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:06.590013 13736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:06.592379 13736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6: Bootstrap starting.
I20260928 12:31:06.592955 13736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:06.593185 13736 log.cc:824] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:06.593884 13736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6: No bootstrap required, opened a new log
I20260928 12:31:06.594290 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 13669
I20260928 12:31:06.594399 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:06.594595 8244 external_mini_cluster.cc:1428] Setting key ac9461595451f65d1386b48b402ae895
I20260928 12:31:06.595942 13736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c6d0624cb44ab291d96c7e64768ee6" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42273 } }
I20260928 12:31:06.596107 13736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:06.596143 13736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c6d0624cb44ab291d96c7e64768ee6, State: Initialized, Role: FOLLOWER
I20260928 12:31:06.596302 13736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c6d0624cb44ab291d96c7e64768ee6" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42273 } }
I20260928 12:31:06.596386 13736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:06.596431 13736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:06.596484 13736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:06.597075 13736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c6d0624cb44ab291d96c7e64768ee6" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42273 } }
I20260928 12:31:06.597167 13736 leader_election.cc:302] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35c6d0624cb44ab291d96c7e64768ee6; no voters:
I20260928 12:31:06.597347 13736 leader_election.cc:288] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:06.597378 13741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:06.597479 13741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [term 1 LEADER]: Becoming Leader. State: Replica: 35c6d0624cb44ab291d96c7e64768ee6, State: Running, Role: LEADER
I20260928 12:31:06.597564 13741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c6d0624cb44ab291d96c7e64768ee6" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42273 } }
I20260928 12:31:06.597625 13736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:06.598279 13742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35c6d0624cb44ab291d96c7e64768ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c6d0624cb44ab291d96c7e64768ee6" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42273 } } }
I20260928 12:31:06.598368 13742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:06.598560 13751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:06.599071 13751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:06.599040 13743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35c6d0624cb44ab291d96c7e64768ee6. Latest consensus state: current_term: 1 leader_uuid: "35c6d0624cb44ab291d96c7e64768ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c6d0624cb44ab291d96c7e64768ee6" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42273 } } }
I20260928 12:31:06.599717 13743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:06.601217 13751 catalog_manager.cc:1347] Generated new cluster ID: 033f1d03a17b4f2eb22c4abd4bcbce98
I20260928 12:31:06.601274 13751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:31:06.602290 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:06.595111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.8.13.62:42273)
0928 12:31:06.595305 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:06.595314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:06.595340 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:31:06.595907 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:06.595913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:06.595973 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:06.596090 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:06.596320 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.597698 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.598766 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.599423 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.599765 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:06.600624 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:06.600629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:06.600629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:06.600845 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:06.600901 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:06.600932 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:06.601019 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:06.601945 (+ 926us) server_negotiation.cc:299] Negotiation successful
0928 12:31:06.602052 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260928 12:31:06.603529 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:31:06.606361 13751 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:06.606839 13751 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:06.610788 13751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35c6d0624cb44ab291d96c7e64768ee6: Generated new TSK 0
I20260928 12:31:06.610952 13751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:31:07.076303 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:06.617556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.8.13.62:42273)
0928 12:31:06.617602 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:06.617604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:06.617669 (+ 65us) server_negotiation.cc:407] Connection header received
0928 12:31:06.617758 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:06.617759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:06.617768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:06.617878 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:31:06.618008 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.618483 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.619671 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.620120 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.620192 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:06.620385 (+ 193us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:31:07.076055 (+455670us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 12:31:07.076237 (+ 182us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44276: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 12:31:07.076372 13740 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44276: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 12:31:07.076452 13765 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:06.617475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:42273 (local address 127.0.0.1:44276)
0928 12:31:06.617605 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:06.617617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:06.617689 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:06.617879 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:06.617880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:06.617882 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:31:06.617947 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:06.617950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.618560 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.618560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:06.619516 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:06.619519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:06.620201 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:06.620209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:06.620272 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:06.620278 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:31:07.076233 (+455955us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 12:31:07.076359 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.13.62:42273: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260928 12:31:07.076766 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 13669
2026-09-28T12:31:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (643 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:31:07 dist-test-slave-g31s krb5kdc[13777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:31:07 dist-test-slave-g31s krb5kdc[13777](info): set up 2 sockets
Sep 28 12:31:07 dist-test-slave-g31s krb5kdc[13777](info): commencing operation
krb5kdc: starting...
W20260928 12:31:08.237782 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:31:08 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:31:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:31:08.361588 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:43381
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:43381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:08.448109 13790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:08.448387 13790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:08.448426 13790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:08.450174 13790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:08.450230 13790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:08.450249 13790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:08.450266 13790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:08.450304 13790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:08.452060 13790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:43381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:43381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:08.452591 13790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:08.453042 13790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:08.458169 13795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:08.458207 13796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:08.458209 13790 server_base.cc:1034] running on GCE node
W20260928 12:31:08.458169 13798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:08.458637 13790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:08.458937 13790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:08.460124 13790 hybrid_clock.cc:648] HybridClock initialized: now 1790598668460071 us; error 68 us; skew 500 ppm
Sep 28 12:31:08 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598668, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:08.466396 13790 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:31:08.466903 13790 webserver.cc:460] Webserver started at http://127.8.13.62:37241/ using document root <none> and password file <none>
I20260928 12:31:08.467142 13790 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:08.467193 13790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:08.467319 13790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:08.468400 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "1b2b02dc6d3444be91640fc02df0d7b3"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "5e00a09d5186c6e3c01c302a451567a7"
server_key_iv: "1ddc04d48368e683af7ecd3607c835e2"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.468739 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "1b2b02dc6d3444be91640fc02df0d7b3"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "5e00a09d5186c6e3c01c302a451567a7"
server_key_iv: "1ddc04d48368e683af7ecd3607c835e2"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.470152 13790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:31:08.470927 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:08.471096 13790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:31:08.471181 13790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "1b2b02dc6d3444be91640fc02df0d7b3"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "5e00a09d5186c6e3c01c302a451567a7"
server_key_iv: "1ddc04d48368e683af7ecd3607c835e2"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.471244 13790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:08.483078 13790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:08.485570 13790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:08.485715 13790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:08.490185 13790 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:43381
I20260928 12:31:08.490222 13857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:43381 every 8 connection(s)
I20260928 12:31:08.490588 13790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:08.491432 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:08.493692 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3: Bootstrap starting.
I20260928 12:31:08.494355 13858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:08.494622 13858 log.cc:824] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:08.495585 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3: No bootstrap required, opened a new log
I20260928 12:31:08.497256 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 13790
I20260928 12:31:08.497380 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:08.497597 8244 external_mini_cluster.cc:1428] Setting key 742a8ab77bacecc9ea361a006f3f4d8d
I20260928 12:31:08.498273 13858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2b02dc6d3444be91640fc02df0d7b3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43381 } }
I20260928 12:31:08.498492 13858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:08.498526 13858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2b02dc6d3444be91640fc02df0d7b3, State: Initialized, Role: FOLLOWER
I20260928 12:31:08.498660 13858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2b02dc6d3444be91640fc02df0d7b3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43381 } }
I20260928 12:31:08.498744 13858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:08.498775 13858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:08.498814 13858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:08.499967 13858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2b02dc6d3444be91640fc02df0d7b3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43381 } }
I20260928 12:31:08.500092 13858 leader_election.cc:302] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b2b02dc6d3444be91640fc02df0d7b3; no voters:
I20260928 12:31:08.500270 13858 leader_election.cc:288] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:08.500349 13863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:08.500499 13863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [term 1 LEADER]: Becoming Leader. State: Replica: 1b2b02dc6d3444be91640fc02df0d7b3, State: Running, Role: LEADER
I20260928 12:31:08.500537 13858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:08.500597 13863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2b02dc6d3444be91640fc02df0d7b3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43381 } }
I20260928 12:31:08.501765 13865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b2b02dc6d3444be91640fc02df0d7b3. Latest consensus state: current_term: 1 leader_uuid: "1b2b02dc6d3444be91640fc02df0d7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2b02dc6d3444be91640fc02df0d7b3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43381 } } }
I20260928 12:31:08.502123 13864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b2b02dc6d3444be91640fc02df0d7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2b02dc6d3444be91640fc02df0d7b3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43381 } } }
I20260928 12:31:08.502182 13864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:08.501893 13865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3 [sys.catalog]: This master's current role is: LEADER
W20260928 12:31:08.502452 13878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:31:08.502516 13878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:31:08.502595 13880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:08.502791 13880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:08.504647 13880 catalog_manager.cc:1347] Generated new cluster ID: d13bfcca41ac4263b5803345eaea8ad4
I20260928 12:31:08.504712 13880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:31:08 dist-test-slave-g31s krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:08.508657 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:08.498198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.8.13.62:43381)
0928 12:31:08.498367 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:08.498378 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:08.498419 (+ 41us) server_negotiation.cc:407] Connection header received
0928 12:31:08.498538 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:08.498545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:08.498605 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:08.498748 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:08.499405 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:08.500802 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:08.501845 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:08.502489 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:08.502611 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:08.505508 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:08.505514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:08.505514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:08.505528 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:08.507187 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:08.507540 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:08.507543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:08.507544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:08.507644 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:08.507817 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:08.507818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:08.507818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:08.508075 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:08.508218 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:08.508313 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 12:31:08.508451 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260928 12:31:08.512223 13880 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:08.512768 13880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:08.515468 13880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b2b02dc6d3444be91640fc02df0d7b3: Generated new TSK 0
I20260928 12:31:08.515625 13880 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:31:08.545473 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43381
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:08.630545 13884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:08.630766 13884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:08.630792 13884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:08.632061 13884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:08.632117 13884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:08.632146 13884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:08.633765 13884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:08.634048 13884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:08.634299 13884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:08.641263 13890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:08.641284 13892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:08.641218 13889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:08.641909 13884 server_base.cc:1034] running on GCE node
I20260928 12:31:08.642162 13884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:08.642418 13884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:08.643590 13884 hybrid_clock.cc:648] HybridClock initialized: now 1790598668643575 us; error 32 us; skew 500 ppm
Sep 28 12:31:08 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598668, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:08.646231 13884 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:31:08.646656 13884 webserver.cc:460] Webserver started at http://127.8.13.1:43603/ using document root <none> and password file <none>
I20260928 12:31:08.646862 13884 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:08.646919 13884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:08.647042 13884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:08.648146 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "35bc3bacb2224b6393ad9b74281731d3"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "34f784749ab53ba9c8e5bbdcf4143644"
server_key_iv: "8a5f3e60d8913ea993646fd4c9ce0ce5"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.648492 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "35bc3bacb2224b6393ad9b74281731d3"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "34f784749ab53ba9c8e5bbdcf4143644"
server_key_iv: "8a5f3e60d8913ea993646fd4c9ce0ce5"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.650012 13884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:31:08.651221 13899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:08.651433 13884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:08.651535 13884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "35bc3bacb2224b6393ad9b74281731d3"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "34f784749ab53ba9c8e5bbdcf4143644"
server_key_iv: "8a5f3e60d8913ea993646fd4c9ce0ce5"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.651610 13884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:08.664453 13884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:08.666806 13884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:08.666963 13884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:08.667202 13884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:08.667522 13884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:08.667559 13884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:08.667593 13884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:08.667618 13884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:08.673228 13884 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:35917
I20260928 12:31:08.673305 14012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:35917 every 8 connection(s)
I20260928 12:31:08.673705 13884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:31:08 dist-test-slave-g31s krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598668, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:08.681357 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 13884
I20260928 12:31:08.681465 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:31:08.681665 8244 external_mini_cluster.cc:1428] Setting key 1eddae5eb09f1183e2cf91f6de3e1c6e
I20260928 12:31:08.684907 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:08.674590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:58955 (local address 127.8.13.62:43381)
0928 12:31:08.674680 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:08.674682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:08.674825 (+ 143us) server_negotiation.cc:407] Connection header received
0928 12:31:08.675812 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:08.675813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:08.675825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:08.675892 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:08.676548 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:08.677061 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:08.678240 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:08.678459 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:08.678525 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:08.680884 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:08.680886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:08.680886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:08.680898 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:08.682914 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:08.683591 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:08.683593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:08.683593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:08.683681 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:08.684089 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:08.684090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:08.684090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:08.684222 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:08.684285 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:08.684734 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 12:31:08.684831 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 12:31:08.685315 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:08.674540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:43381 (local address 127.8.13.1:58955)
0928 12:31:08.674696 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:08.674741 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:08.675690 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:08.675949 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:08.675953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:08.676229 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:08.676432 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:08.676437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:08.677180 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:08.677181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:08.678152 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:08.678155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:08.678565 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:08.678566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:08.678686 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:08.679240 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:08.679248 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:08.680757 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:08.683058 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:08.683092 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:08.683096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:08.683491 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:08.683790 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:08.683791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:08.683791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:08.683986 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:08.684293 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:08.684300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:08.684533 (+ 233us) client_negotiation.cc:769] Sending connection context
0928 12:31:08.684671 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 12:31:08.684905 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260928 12:31:08.686122 14013 heartbeater.cc:347] Connected to a master server at 127.8.13.62:43381
I20260928 12:31:08.686240 14013 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:08.686466 14013 heartbeater.cc:511] Master 127.8.13.62:43381 requested a full tablet report, sending...
I20260928 12:31:08.687073 13822 ts_manager.cc:194] Registered new tserver with Master: 35bc3bacb2224b6393ad9b74281731d3 (127.8.13.1:35917)
I20260928 12:31:08.687894 13822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:58955
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:31:08.712774 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43381
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:08.797204 14018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:08.797434 14018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:08.797467 14018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:08.798866 14018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:08.798916 14018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:08.798947 14018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:08.800665 14018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:08.800905 14018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:08.801154 14018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:08.806397 14024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:08.806434 14018 server_base.cc:1034] running on GCE node
W20260928 12:31:08.806401 14026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:08.806437 14023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:08.806849 14018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:08.807085 14018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:08.808249 14018 hybrid_clock.cc:648] HybridClock initialized: now 1790598668808227 us; error 37 us; skew 500 ppm
Sep 28 12:31:08 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598668, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:08.810882 14018 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:31:08.811398 14018 webserver.cc:460] Webserver started at http://127.8.13.2:38599/ using document root <none> and password file <none>
I20260928 12:31:08.811635 14018 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:08.811679 14018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:08.811831 14018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:08.812947 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "5790e68469874f4a9806a9851e0ad8bc"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "cdeb66a82a9b990b15ade4c12428e430"
server_key_iv: "21e4b0244c7d0ee090e62f3bcdb136c5"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.813264 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "5790e68469874f4a9806a9851e0ad8bc"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "cdeb66a82a9b990b15ade4c12428e430"
server_key_iv: "21e4b0244c7d0ee090e62f3bcdb136c5"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.814651 14018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:31:08.815433 14033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:08.815605 14018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:08.815683 14018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "5790e68469874f4a9806a9851e0ad8bc"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "cdeb66a82a9b990b15ade4c12428e430"
server_key_iv: "21e4b0244c7d0ee090e62f3bcdb136c5"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.815752 14018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:08.832651 14018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:08.834965 14018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:08.835093 14018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:08.835327 14018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:08.835629 14018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:08.835664 14018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:08.835695 14018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:08.835717 14018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:08.842376 14018 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:34209
I20260928 12:31:08.842486 14146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:34209 every 8 connection(s)
I20260928 12:31:08.842790 14018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:31:08.848197 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 14018
I20260928 12:31:08.848338 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:31:08.848543 8244 external_mini_cluster.cc:1428] Setting key e7c14c8200b1b3213f87ceeb0e02ce1a
Sep 28 12:31:08 dist-test-slave-g31s krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598668, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:08.853490 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:08.843601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:36451 (local address 127.8.13.62:43381)
0928 12:31:08.843695 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:08.843697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:08.843856 (+ 159us) server_negotiation.cc:407] Connection header received
0928 12:31:08.844847 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:08.844849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:08.844866 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:08.844954 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:08.845688 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:08.846271 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:08.847502 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:08.847712 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:08.847799 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:08.850329 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:08.850331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:08.850331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:08.850340 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:08.851943 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:08.852461 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:08.852462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:08.852463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:08.852538 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:08.852765 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:08.852765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:08.852766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:08.852927 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:08.853002 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:08.853284 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 12:31:08.853391 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 12:31:08.853607 14149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:08.843557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:43381 (local address 127.8.13.2:36451)
0928 12:31:08.843717 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:08.843759 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:08.844684 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:08.845038 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:08.845041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:08.845328 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:08.845543 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:08.845547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:08.846408 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:08.846409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:08.847374 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:08.847377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:08.847821 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:08.847822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:08.847977 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:08.848638 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:08.848650 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:08.850225 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:08.852104 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:08.852113 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:08.852116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:08.852384 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:08.852602 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:08.852602 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:08.852602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:08.852707 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:08.852997 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:08.853003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:08.853150 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 12:31:08.853242 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 12:31:08.853387 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260928 12:31:08.854152 14147 heartbeater.cc:347] Connected to a master server at 127.8.13.62:43381
I20260928 12:31:08.854250 14147 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:08.854431 14147 heartbeater.cc:511] Master 127.8.13.62:43381 requested a full tablet report, sending...
I20260928 12:31:08.854871 13822 ts_manager.cc:194] Registered new tserver with Master: 5790e68469874f4a9806a9851e0ad8bc (127.8.13.2:34209)
I20260928 12:31:08.855243 13822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:36451
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:31:08.878211 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43381
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:08.961138 14152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:08.961426 14152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:08.961462 14152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:08.962996 14152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:08.963058 14152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:08.963101 14152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:08.964798 14152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:08.965092 14152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:08.965323 14152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:08.970681 14158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:08.970700 14160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:08.970734 14157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:08.971031 14152 server_base.cc:1034] running on GCE node
I20260928 12:31:08.971302 14152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:08.971576 14152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:08.972751 14152 hybrid_clock.cc:648] HybridClock initialized: now 1790598668972738 us; error 33 us; skew 500 ppm
Sep 28 12:31:08 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598668, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:08.975539 14152 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:31:08.976051 14152 webserver.cc:460] Webserver started at http://127.8.13.3:40359/ using document root <none> and password file <none>
I20260928 12:31:08.976274 14152 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:08.976326 14152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:08.976466 14152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:08.977579 14152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "0d8bb402955140948ee03a911224b537"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "a8acf56da03ce77848887a6d6533bc82"
server_key_iv: "86e3d79950b6a113f5c612e4dc7d4b0c"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.977972 14152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "0d8bb402955140948ee03a911224b537"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "a8acf56da03ce77848887a6d6533bc82"
server_key_iv: "86e3d79950b6a113f5c612e4dc7d4b0c"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.979297 14152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:08.980086 14167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:08.980261 14152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:31:08.980347 14152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "0d8bb402955140948ee03a911224b537"
format_stamp: "Formatted at 2026-09-28 12:31:08 on dist-test-slave-g31s"
server_key: "a8acf56da03ce77848887a6d6533bc82"
server_key_iv: "86e3d79950b6a113f5c612e4dc7d4b0c"
server_key_version: "encryptionkey@0"
I20260928 12:31:08.980413 14152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:08.992971 14152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:08.995213 14152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:08.995350 14152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:08.995579 14152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:08.995882 14152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:08.995915 14152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:08.995946 14152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:08.995967 14152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:09.001778 14152 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:41123
I20260928 12:31:09.001861 14280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:41123 every 8 connection(s)
I20260928 12:31:09.002228 14152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:31:09.003542 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 14152
I20260928 12:31:09.003634 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:31:09.003835 8244 external_mini_cluster.cc:1428] Setting key 8286df478a16cd5262a250474f1996a8
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598668, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:09.011801 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:09.002948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:45647 (local address 127.8.13.62:43381)
0928 12:31:09.002991 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:09.002993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:09.003190 (+ 197us) server_negotiation.cc:407] Connection header received
0928 12:31:09.004321 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:09.004322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:09.004334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:09.004375 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:09.005164 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:09.006004 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:09.007054 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:09.007256 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:09.007320 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:09.009271 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:09.009272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:09.009273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:09.009279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:09.010462 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:09.010876 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:09.010877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:09.010878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:09.010938 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:09.011148 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:09.011149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:09.011149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:09.011274 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:09.011331 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:09.011621 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 12:31:09.011708 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:31:09.011929 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:09.002936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:43381 (local address 127.8.13.3:45647)
0928 12:31:09.003083 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:09.003121 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:09.004233 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:09.004457 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:09.004470 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:09.004813 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:09.005016 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:09.005021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:09.006104 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:09.006105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:09.006950 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:09.006952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:09.007325 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:09.007326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:09.007422 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:09.007875 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:09.007884 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:09.009190 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:09.010544 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:09.010550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:09.010552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:09.010784 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:09.010996 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:09.010997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:09.010997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:09.011090 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:09.011364 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:09.011370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:09.011503 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 12:31:09.011597 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 12:31:09.011730 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260928 12:31:09.012379 14281 heartbeater.cc:347] Connected to a master server at 127.8.13.62:43381
I20260928 12:31:09.012480 14281 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:09.012658 14281 heartbeater.cc:511] Master 127.8.13.62:43381 requested a full tablet report, sending...
I20260928 12:31:09.013338 13822 ts_manager.cc:194] Registered new tserver with Master: 0d8bb402955140948ee03a911224b537 (127.8.13.3:41123)
I20260928 12:31:09.013716 13822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:45647
I20260928 12:31:09.015427 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/test-admin.keytab.
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:09.030227 8244 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 12:31:09.031891 14288 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:09.031126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.8.13.1:35917)
0928 12:31:09.031279 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:09.031284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:09.031295 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:31:09.031323 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:09.031325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:09.031339 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:09.031407 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:09.031809 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57168: BlockingRecv error: recv got EOF from 127.0.0.1:57168 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
W20260928 12:31:09.032008 8244 init.cc:244] Error while opening credential cache '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5cc)
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:09.032995 8244 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:09.033910 8244 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:31:09.042886 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:09.034480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.8.13.1:35917)
0928 12:31:09.034696 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:09.034698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:09.034762 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:31:09.034790 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:09.034791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:09.034803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:09.034863 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:09.035306 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:09.036079 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:09.036942 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:09.037444 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:09.037556 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:09.039901 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:09.039916 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:09.039916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:09.039936 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:09.041676 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:09.042130 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:09.042133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:09.042134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:09.042190 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:09.042325 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:09.042325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:09.042325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:09.042476 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:09.042595 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:09.042704 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 12:31:09.042776 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
W20260928 12:31:09.044726 14288 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:09.044242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.8.13.1:35917)
0928 12:31:09.044280 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:09.044282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:09.044290 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:09.044340 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:09.044340 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:09.044350 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:09.044391 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:09.044676 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57186: BlockingRecv error: recv got EOF from 127.0.0.1:57186 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 12:31:09.044859 8244 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5cc' of type FILE
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:09.046240 8244 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:09.047362 8244 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:31:09.054730 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:09.047936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.8.13.1:35917)
0928 12:31:09.047998 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:09.048000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:09.048009 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:09.048067 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:09.048068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:09.048078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:09.048123 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:09.048479 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:09.048976 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:09.049633 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:09.049875 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:09.049938 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:09.052057 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:09.052059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:09.052059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:09.052068 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:09.053349 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:09.053778 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:09.053781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:09.053782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:09.053959 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:09.054267 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:09.054268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:09.054269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:09.054400 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:09.054507 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:09.054590 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 12:31:09.054660 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 12:31:09.056365 14288 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:09.055819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.8.13.1:35917)
0928 12:31:09.055858 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:09.055860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:09.055902 (+ 42us) server_negotiation.cc:407] Connection header received
0928 12:31:09.055973 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:09.055974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:09.055984 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:09.056031 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:09.056312 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57204: BlockingRecv error: recv got EOF from 127.0.0.1:57204 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 12:31:09.056437 8244 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598640018661-8244-0/krb5kdc/krb5cc' of type FILE
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:09.057693 8244 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:09.059001 8244 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:31:09.066805 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:09.059743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.8.13.1:35917)
0928 12:31:09.059800 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:09.059802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:09.059814 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:31:09.059890 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:09.059891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:09.059901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:09.059953 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:09.060343 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:09.060913 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:09.061626 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:09.061868 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:09.061924 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:09.064230 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:09.064231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:09.064232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:09.064241 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:09.065725 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:09.066103 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:09.066104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:09.066105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:09.066159 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:09.066317 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:09.066318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:09.066318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:09.066425 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:09.066508 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:09.066624 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 12:31:09.066704 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:31:09.067576 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 13884
I20260928 12:31:09.071930 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 14018
I20260928 12:31:09.076185 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 14152
I20260928 12:31:09.081568 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 13790
2026-09-28T12:31:09Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1826 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[14301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[14301](info): set up 2 sockets
Sep 28 12:31:09 dist-test-slave-g31s krb5kdc[14301](info): commencing operation
krb5kdc: starting...
W20260928 12:31:10.049997 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:31:10 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:31:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.62.keytab.
Entry for principal oryx/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.62.keytab.
I20260928 12:31:10.178046 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45429
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.62.keytab
--principal=oryx/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:10.266076 14314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:10.266372 14314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:10.266402 14314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:10.267872 14314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:10.267918 14314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:10.267933 14314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:10.267946 14314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:10.267956 14314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:10.269670 14314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.62.keytab
--principal=oryx/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:10.269965 14314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:10.270221 14314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:10.276180 14320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:10.276196 14319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:10.276353 14314 server_base.cc:1034] running on GCE node
W20260928 12:31:10.276196 14322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:10.276809 14314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:10.277156 14314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:10.278314 14314 hybrid_clock.cc:648] HybridClock initialized: now 1790598670278298 us; error 37 us; skew 500 ppm
Sep 28 12:31:10 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:10.281131 14314 init.cc:377] Logged in from keytab as oryx/127.8.13.62@KRBTEST.COM (short username oryx)
I20260928 12:31:10.281992 14314 webserver.cc:460] Webserver started at http://127.8.13.62:34587/ using document root <none> and password file <none>
I20260928 12:31:10.282213 14314 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:10.282263 14314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:10.282395 14314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:10.283944 14314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "2726055b354a40239b82c2c722552b5f"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "508dfd9fe8d972e070e9050aa5266bf0"
server_key_iv: "cfaf31cd47e02b867e390354d38f5c97"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.284390 14314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "2726055b354a40239b82c2c722552b5f"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "508dfd9fe8d972e070e9050aa5266bf0"
server_key_iv: "cfaf31cd47e02b867e390354d38f5c97"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.285784 14314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 12:31:10.286723 14329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:10.286954 14314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:10.287053 14314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "2726055b354a40239b82c2c722552b5f"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "508dfd9fe8d972e070e9050aa5266bf0"
server_key_iv: "cfaf31cd47e02b867e390354d38f5c97"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.287127 14314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:10.318996 14314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:10.321168 14314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:10.321313 14314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:10.325030 14314 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:45429
I20260928 12:31:10.325093 14381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:45429 every 8 connection(s)
I20260928 12:31:10.325416 14314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:10.326318 14382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:10.328625 14382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f: Bootstrap starting.
I20260928 12:31:10.329191 14382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:10.329455 14382 log.cc:824] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:10.330216 14382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f: No bootstrap required, opened a new log
I20260928 12:31:10.331894 14382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } }
I20260928 12:31:10.332056 14382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:10.332103 14382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2726055b354a40239b82c2c722552b5f, State: Initialized, Role: FOLLOWER
I20260928 12:31:10.332250 14382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } }
I20260928 12:31:10.332325 14382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:10.332357 14382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:10.332405 14382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:10.333266 14382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } }
I20260928 12:31:10.333369 14382 leader_election.cc:302] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2726055b354a40239b82c2c722552b5f; no voters:
I20260928 12:31:10.333544 14382 leader_election.cc:288] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:10.333623 14385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:10.333792 14385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 1 LEADER]: Becoming Leader. State: Replica: 2726055b354a40239b82c2c722552b5f, State: Running, Role: LEADER
I20260928 12:31:10.333905 14382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:10.333932 14385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } }
I20260928 12:31:10.334300 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 14314
I20260928 12:31:10.334352 14386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } }
I20260928 12:31:10.334410 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:10.334419 14386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:10.334554 14387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2726055b354a40239b82c2c722552b5f. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } }
I20260928 12:31:10.334597 14387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:10.334611 8244 external_mini_cluster.cc:1428] Setting key 7aa7d7b5c2f358ca5ac32f208f0c41da
I20260928 12:31:10.335072 14392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:10.335602 14392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:10.337307 14392 catalog_manager.cc:1347] Generated new cluster ID: ded15d47e7974d518d3d52c27a1cda2a
I20260928 12:31:10.337376 14392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:31:10 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
I20260928 12:31:10.347405 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:10.335199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.8.13.62:45429)
0928 12:31:10.335466 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:10.335478 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:10.335512 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:31:10.335640 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:10.335648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:10.335708 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:10.335848 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:10.336540 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.337762 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.338524 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.338980 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.339078 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:10.342885 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:10.342892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:10.342892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:10.342909 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:10.345115 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:10.345621 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:10.345625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:10.345625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:10.345747 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:10.345997 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:10.345998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:10.345998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:10.346330 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:10.346518 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:10.346773 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 12:31:10.346986 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
I20260928 12:31:10.377894 14392 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:10.378695 14392 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:10.384786 14392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f: Generated new TSK 0
I20260928 12:31:10.385069 14392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab.
Entry for principal oryx/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab.
I20260928 12:31:10.417045 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45429
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab
--principal=oryx/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:10.508630 14408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:10.508858 14408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:10.508885 14408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:10.510186 14408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:10.510236 14408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:10.510280 14408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:10.511845 14408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab
--principal=oryx/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:10.512090 14408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:10.512318 14408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:10.517517 14416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:10.517558 14408 server_base.cc:1034] running on GCE node
W20260928 12:31:10.517513 14413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:10.517560 14414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:10.517973 14408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:10.518260 14408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:10.519424 14408 hybrid_clock.cc:648] HybridClock initialized: now 1790598670519404 us; error 30 us; skew 500 ppm
Sep 28 12:31:10 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:10.522181 14408 init.cc:377] Logged in from keytab as oryx/127.8.13.1@KRBTEST.COM (short username oryx)
I20260928 12:31:10.522665 14408 webserver.cc:460] Webserver started at http://127.8.13.1:42007/ using document root <none> and password file <none>
I20260928 12:31:10.522853 14408 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:10.522893 14408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:10.522986 14408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:10.524046 14408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "b8acc77419c94659863723a919a041f3"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "74941ab07bd18e5c848afc13c9682d66"
server_key_iv: "25e6ae7d112f1f9923d3e42b132654c7"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.524362 14408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "b8acc77419c94659863723a919a041f3"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "74941ab07bd18e5c848afc13c9682d66"
server_key_iv: "25e6ae7d112f1f9923d3e42b132654c7"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.525662 14408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:31:10.526648 14423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:10.526875 14408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:10.526966 14408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "b8acc77419c94659863723a919a041f3"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "74941ab07bd18e5c848afc13c9682d66"
server_key_iv: "25e6ae7d112f1f9923d3e42b132654c7"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.527048 14408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:10.547760 14408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:10.550082 14408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:10.550231 14408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:10.550472 14408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:10.550781 14408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:10.550815 14408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:10.550845 14408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:10.550868 14408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:10.556842 14408 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:39495
I20260928 12:31:10.556895 14536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:39495 every 8 connection(s)
I20260928 12:31:10.557224 14408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:31:10.562729 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 14408
I20260928 12:31:10.562850 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:31:10.563182 8244 external_mini_cluster.cc:1428] Setting key 5ebe309a51fba476aea0d639e342074c
Sep 28 12:31:10 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
I20260928 12:31:10.569229 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:10.558015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:47033 (local address 127.8.13.62:45429)
0928 12:31:10.558082 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:10.558084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:10.558304 (+ 220us) server_negotiation.cc:407] Connection header received
0928 12:31:10.558840 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:10.558841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:10.558854 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:10.558922 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:10.560281 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.560803 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.561916 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.562132 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.562191 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:10.565402 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:10.565404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:10.565405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:10.565413 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:10.567022 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:10.567631 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:10.567633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:10.567634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:10.567725 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:10.568324 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:10.568325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:10.568325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:10.568490 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:10.568575 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:10.569004 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 12:31:10.569118 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:31:10.569443 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:10.558002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.1:47033)
0928 12:31:10.558179 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:10.558219 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:10.558729 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:10.559010 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:10.559015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:10.559905 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:10.560137 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:10.560143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.560904 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.560905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:10.561787 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:10.561790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.562215 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.562215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:10.562356 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:10.563130 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:10.563145 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:10.565251 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:10.567163 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:10.567176 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:10.567179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:10.567497 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:10.567856 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:10.567858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:10.567858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:10.568214 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:10.568603 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:10.568612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:10.568804 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 12:31:10.568948 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 12:31:10.569160 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260928 12:31:10.570171 14537 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45429
I20260928 12:31:10.570328 14537 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:10.570590 14537 heartbeater.cc:511] Master 127.8.13.62:45429 requested a full tablet report, sending...
I20260928 12:31:10.571198 14346 ts_manager.cc:194] Registered new tserver with Master: b8acc77419c94659863723a919a041f3 (127.8.13.1:39495)
I20260928 12:31:10.571810 14346 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:47033
WARNING: no policy specified for oryx/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab.
Entry for principal oryx/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab.
I20260928 12:31:10.594071 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45429
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab
--principal=oryx/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:10.680536 14542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:10.680781 14542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:10.680809 14542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:10.682199 14542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:10.682262 14542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:10.682296 14542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:10.684080 14542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab
--principal=oryx/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:10.684367 14542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:10.684607 14542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:10.690227 14547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:10.690351 14550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:10.690349 14548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:10.690543 14542 server_base.cc:1034] running on GCE node
I20260928 12:31:10.690836 14542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:10.691103 14542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:10.692276 14542 hybrid_clock.cc:648] HybridClock initialized: now 1790598670692248 us; error 41 us; skew 500 ppm
Sep 28 12:31:10 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:10.695578 14542 init.cc:377] Logged in from keytab as oryx/127.8.13.2@KRBTEST.COM (short username oryx)
I20260928 12:31:10.696425 14542 webserver.cc:460] Webserver started at http://127.8.13.2:41733/ using document root <none> and password file <none>
I20260928 12:31:10.696671 14542 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:10.696718 14542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:10.696851 14542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:10.698009 14542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "c33e75f6c55a4f39ae62d0547553141f"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "acf815b787770a1177d9579c6abd96d1"
server_key_iv: "a5428a417cfad92d3426080c9f55309b"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.698350 14542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "c33e75f6c55a4f39ae62d0547553141f"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "acf815b787770a1177d9579c6abd96d1"
server_key_iv: "a5428a417cfad92d3426080c9f55309b"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.699954 14542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:10.701254 14557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:10.701475 14542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:10.701573 14542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "c33e75f6c55a4f39ae62d0547553141f"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "acf815b787770a1177d9579c6abd96d1"
server_key_iv: "a5428a417cfad92d3426080c9f55309b"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.701650 14542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:10.722723 14542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:10.724666 14542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:10.724783 14542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:10.724985 14542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:10.725261 14542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:10.725296 14542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:10.725318 14542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:10.725335 14542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:10.730885 14542 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:44661
I20260928 12:31:10.730937 14670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:44661 every 8 connection(s)
I20260928 12:31:10.731251 14542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:31:10 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
I20260928 12:31:10.739765 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 14542
I20260928 12:31:10.739882 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:31:10.740080 8244 external_mini_cluster.cc:1428] Setting key 86d23f9dad5d203b5df37db64097bcfb
I20260928 12:31:10.741570 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:10.731966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:46589 (local address 127.8.13.62:45429)
0928 12:31:10.732028 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:10.732030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:10.732232 (+ 202us) server_negotiation.cc:407] Connection header received
0928 12:31:10.732720 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:10.732721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:10.732731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:10.732791 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:10.733789 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.734449 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.735601 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.735804 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.735874 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:10.738290 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:10.738291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:10.738291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:10.738298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:10.739744 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:10.740298 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:10.740300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:10.740300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:10.740402 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:10.740711 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:10.740712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:10.740713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:10.740864 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:10.740948 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:10.741411 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 12:31:10.741501 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:31:10.741827 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:10.731928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.2:46589)
0928 12:31:10.732110 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:10.732151 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:10.732639 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:10.732850 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:10.732854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:10.733436 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:10.733632 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:10.733638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.734563 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.734564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:10.735496 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:10.735499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.735914 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.735915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:10.736056 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:10.736623 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:10.736632 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:10.738159 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:10.739864 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:10.739879 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:10.739881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:10.740217 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:10.740490 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:10.740491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:10.740491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:10.740636 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:10.740982 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:10.740990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:10.741210 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 12:31:10.741353 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 12:31:10.741563 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260928 12:31:10.742300 14671 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45429
I20260928 12:31:10.742401 14671 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:10.742594 14671 heartbeater.cc:511] Master 127.8.13.62:45429 requested a full tablet report, sending...
I20260928 12:31:10.742988 14346 ts_manager.cc:194] Registered new tserver with Master: c33e75f6c55a4f39ae62d0547553141f (127.8.13.2:44661)
I20260928 12:31:10.743363 14346 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:46589
WARNING: no policy specified for oryx/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab.
Entry for principal oryx/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab.
I20260928 12:31:10.770283 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45429
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab
--principal=oryx/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:10.858784 14676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:10.859028 14676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:10.859066 14676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:10.860378 14676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:10.860442 14676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:10.860484 14676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:10.862195 14676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab
--principal=oryx/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:10.862483 14676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:10.862715 14676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:10.868013 14682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:10.868081 14681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:10.868285 14676 server_base.cc:1034] running on GCE node
W20260928 12:31:10.868031 14684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:10.868643 14676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:10.868918 14676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:10.870141 14676 hybrid_clock.cc:648] HybridClock initialized: now 1790598670870124 us; error 33 us; skew 500 ppm
Sep 28 12:31:10 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:10.873487 14676 init.cc:377] Logged in from keytab as oryx/127.8.13.3@KRBTEST.COM (short username oryx)
I20260928 12:31:10.874074 14676 webserver.cc:460] Webserver started at http://127.8.13.3:42995/ using document root <none> and password file <none>
I20260928 12:31:10.874317 14676 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:10.874373 14676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:10.874506 14676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:10.875630 14676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "8b411c034e084e3a9e811cb2aceea760"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "2cbc493d61df365cda8c85e002236180"
server_key_iv: "ff0416158305259a36c1ed31606fd3a4"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.876019 14676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "8b411c034e084e3a9e811cb2aceea760"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "2cbc493d61df365cda8c85e002236180"
server_key_iv: "ff0416158305259a36c1ed31606fd3a4"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.877506 14676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:31:10.878408 14691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:10.878578 14676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:31:10.878661 14676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "8b411c034e084e3a9e811cb2aceea760"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "2cbc493d61df365cda8c85e002236180"
server_key_iv: "ff0416158305259a36c1ed31606fd3a4"
server_key_version: "encryptionkey@0"
I20260928 12:31:10.878734 14676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:10.890921 14676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:10.893781 14676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:10.894021 14676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:10.894322 14676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:10.894761 14676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:10.894804 14676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:10.894840 14676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:10.894863 14676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:10.901542 14676 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:46869
I20260928 12:31:10.901594 14804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:46869 every 8 connection(s)
I20260928 12:31:10.902016 14676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:31:10.906528 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 14676
I20260928 12:31:10.906644 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:31:10.906814 8244 external_mini_cluster.cc:1428] Setting key 069663174bf51c76f0a6afca28094baa
Sep 28 12:31:10 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
I20260928 12:31:10.911919 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:10.902850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:47963 (local address 127.8.13.62:45429)
0928 12:31:10.902905 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:10.902908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:10.903097 (+ 189us) server_negotiation.cc:407] Connection header received
0928 12:31:10.903587 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:10.903589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:10.903600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:10.903663 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:10.904705 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.905204 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.906397 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.906637 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.906765 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:10.908974 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:10.908975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:10.908975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:10.908984 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:10.910567 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:10.910986 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:10.910988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:10.910988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:10.911044 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:10.911263 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:10.911263 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:10.911264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:10.911387 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:10.911442 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:10.911726 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 12:31:10.911814 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:31:10.912171 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:10.902791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.3:47963)
0928 12:31:10.902963 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:10.903011 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:10.903495 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:10.903734 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:10.903738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:10.904390 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:10.904609 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:10.904615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.905319 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.905320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:10.906224 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:10.906227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:10.906806 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:10.906808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:10.906964 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:10.907492 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:10.907501 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:10.908871 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:10.910676 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:10.910683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:10.910686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:10.910919 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:10.911099 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:10.911100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:10.911100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:10.911207 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:10.911442 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:10.911447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:10.911603 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 12:31:10.911694 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 12:31:10.911864 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260928 12:31:10.912858 14805 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45429
I20260928 12:31:10.912976 14805 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:10.913233 14805 heartbeater.cc:511] Master 127.8.13.62:45429 requested a full tablet report, sending...
I20260928 12:31:10.913717 14346 ts_manager.cc:194] Registered new tserver with Master: 8b411c034e084e3a9e811cb2aceea760 (127.8.13.3:46869)
I20260928 12:31:10.914197 14346 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:47963
I20260928 12:31:10.918824 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.13.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.61.keytab.
Entry for principal oryx/127.8.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.61.keytab.
I20260928 12:31:10.946142 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.61:34599
--webserver_interface=127.8.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.61.keytab
--principal=oryx/127.8.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:45429,127.8.13.61:34599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:11.031811 14810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:11.032042 14810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:11.032064 14810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:11.033221 14810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:11.033262 14810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:11.033275 14810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:11.033286 14810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:11.033298 14810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:11.034891 14810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:45429,127.8.13.61:34599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.61.keytab
--principal=oryx/127.8.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.61:34599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
W20260928 12:31:11.035030 14810 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.13.62:45429,127.8.13.61:34599'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 12:31:11.035187 14810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:11.035411 14810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:11.042122 14815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.042234 14816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.042122 14818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:11.042547 14810 server_base.cc:1034] running on GCE node
I20260928 12:31:11.042841 14810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:11.043179 14810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:11.044387 14810 hybrid_clock.cc:648] HybridClock initialized: now 1790598671044341 us; error 41 us; skew 500 ppm
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:11.047705 14810 init.cc:377] Logged in from keytab as oryx/127.8.13.61@KRBTEST.COM (short username oryx)
I20260928 12:31:11.048373 14810 webserver.cc:460] Webserver started at http://127.8.13.61:35891/ using document root <none> and password file <none>
I20260928 12:31:11.048643 14810 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:11.048698 14810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:11.048847 14810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:11.050662 14810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data/instance:
uuid: "73a8920c213d4947a6504ea84885e8fb"
format_stamp: "Formatted at 2026-09-28 12:31:11 on dist-test-slave-g31s"
server_key: "bedead754dc66f7f35b6969207ed1c4b"
server_key_iv: "41fc9d988608c8ed743bc1e73809f3a7"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.051170 14810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/wal/instance:
uuid: "73a8920c213d4947a6504ea84885e8fb"
format_stamp: "Formatted at 2026-09-28 12:31:11 on dist-test-slave-g31s"
server_key: "bedead754dc66f7f35b6969207ed1c4b"
server_key_iv: "41fc9d988608c8ed743bc1e73809f3a7"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.052802 14810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:31:11.053964 14825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.054153 14810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:11.054296 14810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/wal
uuid: "73a8920c213d4947a6504ea84885e8fb"
format_stamp: "Formatted at 2026-09-28 12:31:11 on dist-test-slave-g31s"
server_key: "bedead754dc66f7f35b6969207ed1c4b"
server_key_iv: "41fc9d988608c8ed743bc1e73809f3a7"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.054423 14810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:11.070003 14810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:11.072999 14810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:11.073163 14810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:11.077404 14810 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.61:34599
I20260928 12:31:11.077499 14877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.61:34599 every 8 connection(s)
I20260928 12:31:11.077865 14810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
I20260928 12:31:11.078994 14878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:11.080662 14878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:31:11.081478 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 14810
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.61@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.13.61@KRBTEST.COM
I20260928 12:31:11.092638 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.081482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.8.13.62:45429)
0928 12:31:11.081539 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.081541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.081838 (+ 297us) server_negotiation.cc:407] Connection header received
0928 12:31:11.082482 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.082483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.082496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.082545 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.083379 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.084349 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.085848 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.086167 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.086234 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.088616 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.088618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.088619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.088628 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.090678 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.091241 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.091243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.091244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.091362 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.091837 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.091844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.091844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.092092 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.092151 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.092430 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.092531 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:31:11.092810 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.081449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:48860)
0928 12:31:11.081704 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.081744 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.082379 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.082577 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.082580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.082935 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.083262 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.083269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.084503 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.084504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.085709 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.085713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.086258 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.086259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.086436 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.086948 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.086956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.088503 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.090767 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.090773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.090776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.091120 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.091517 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.091518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.091519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.091706 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.092155 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.092159 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.092273 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.092364 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.092551 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":8,"thread_start_us":108,"threads_started":1}
I20260928 12:31:11.092810 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.081984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.8.13.61:34599)
0928 12:31:11.082159 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.082164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.082202 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:31:11.082389 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.082397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.082419 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.082509 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.083378 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.084750 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.085882 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.086377 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.086442 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.089125 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.089134 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.089135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.089142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.091104 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.091629 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.091633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.091633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.091705 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.092015 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.092015 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.092015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.092285 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.092472 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.092579 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.092655 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260928 12:31:11.093506 14878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.61@KRBTEST.COM for oryx/127.8.13.61@KRBTEST.COM
I20260928 12:31:11.100922 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.093772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:34599 (local address 127.0.0.1:57374)
0928 12:31:11.093836 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.093852 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.093897 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.093989 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.093990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.094114 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.094171 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.094172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.094894 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.094896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.095505 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.095508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.096080 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.096082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.096180 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.096715 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.096720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.098082 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.099738 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.099739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.099740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.099920 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.100167 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.100168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.100168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.100240 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.100592 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.100593 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.100639 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.100706 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.100811 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 12:31:11.100984 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.093826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.8.13.61:34599)
0928 12:31:11.093861 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.093862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.093883 (+ 21us) server_negotiation.cc:407] Connection header received
0928 12:31:11.093935 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.093936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.093945 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.093987 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.094253 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.094759 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.095699 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.095957 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.096025 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.098277 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.098279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.098279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.098286 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.099625 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.100013 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.100015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.100015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.100073 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.100358 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.100360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.100360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.100504 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.100566 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.100767 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.100865 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":16}
I20260928 12:31:11.102563 14878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: Bootstrap starting.
I20260928 12:31:11.103377 14878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:11.103609 14878 log.cc:824] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:11.104434 14878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: No bootstrap required, opened a new log
I20260928 12:31:11.106036 14878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } }
I20260928 12:31:11.106204 14878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:11.106227 14878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a8920c213d4947a6504ea84885e8fb, State: Initialized, Role: FOLLOWER
I20260928 12:31:11.106328 14878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } }
I20260928 12:31:11.106616 14878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:11.106577 14884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } } }
I20260928 12:31:11.106664 14884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:31:11.108309 14895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:31:11.108381 14895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:31:11.110018 14810 master_runner.cc:428] Detected that this master 73a8920c213d4947a6504ea84885e8fb is joining an existing cluster
I20260928 12:31:11.110067 14810 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.13.61:34599
I20260928 12:31:11.117620 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.111054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.8.13.61:34599)
0928 12:31:11.111095 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.111096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.111289 (+ 193us) server_negotiation.cc:407] Connection header received
0928 12:31:11.111335 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.111336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.111344 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.111381 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.111765 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.112503 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.113595 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.113896 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.113935 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.114949 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.114950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.114951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.114957 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.116531 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.116902 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.116903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.116903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.116955 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.117115 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.117115 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.117115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.117242 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.117305 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.117445 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.117528 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:31:11.117673 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.111194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.8.13.62:45429)
0928 12:31:11.111237 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.111239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.111248 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:11.111278 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.111278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.111292 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.111349 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.111805 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.112524 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.113304 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.113503 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.113557 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.114770 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.114772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.114772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.114780 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.116350 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.116947 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.116948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.116948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.116999 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.117206 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.117207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.117208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.117325 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.117380 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.117527 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.117599 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:31:11.117835 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.111124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:34599 (local address 127.0.0.1:57388)
0928 12:31:11.111243 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.111251 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.111296 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.111392 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.111393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.111635 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.111701 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.111703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.112588 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.112589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.113501 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.113504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.114011 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.114012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.114073 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.114418 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.114423 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.114867 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.116594 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.116595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.116595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.116838 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.117015 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.117015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.117016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.117066 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.117303 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.117304 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.117352 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.117423 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.117607 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":29,"thread_start_us":75,"threads_started":1}
I20260928 12:31:11.118160 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.110985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:48866)
0928 12:31:11.111132 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.111144 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.111223 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.111401 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.111403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.111673 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.111737 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.111739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.112600 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.112601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.113223 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.113225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.113564 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.113565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.113623 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.114097 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.114102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.114682 (+ 580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.116494 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.116496 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.116496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.116841 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.117057 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.117058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.117058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.117140 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.117373 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.117374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.117417 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.117739 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.117870 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":30,"thread_start_us":84,"threads_started":1}
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.62@KRBTEST.COM for oryx/127.8.13.61@KRBTEST.COM
I20260928 12:31:11.128404 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.120790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57398 (local address 127.8.13.61:34599)
0928 12:31:11.120838 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.120840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.120871 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:31:11.120976 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.120977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.120985 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.121025 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.121576 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.122121 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.123273 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.123465 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.123541 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.125897 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.125899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.125899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.125906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.127205 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.127689 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.127690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.127691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.127754 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.127935 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.127936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.127936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.128046 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.128093 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.128271 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.128336 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:31:11.128492 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.120634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:34599 (local address 127.0.0.1:57398)
0928 12:31:11.120780 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.120820 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.120900 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.121043 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.121047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.121411 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.121492 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.121495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.122194 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.122195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.123172 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.123176 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.123557 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.123558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.123651 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.124164 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.124169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.125789 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.127288 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.127299 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.127302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.127599 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.127819 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.127820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.127820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.127885 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.128095 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.128101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.128161 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.128243 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.128376 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260928 12:31:11.129109 14346 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.13.61:34599: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "73a8920c213d4947a6504ea84885e8fb"
member_type: NON_VOTER
last_known_addr {
host: "127.8.13.61"
port: 34599
}
attrs {
promote: true
}
}
dest_uuid: "2726055b354a40239b82c2c722552b5f"
cas_config_opid_index: -1
I20260928 12:31:11.129456 14346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } }
I20260928 12:31:11.129987 14905 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 73a8920c213d4947a6504ea84885e8fb (127.8.13.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } }
I20260928 12:31:11.130263 14852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:11.131002 14905 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.13.61:34599
I20260928 12:31:11.131013 14852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Refusing update from remote peer 2726055b354a40239b82c2c722552b5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 12:31:11.131083 14905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } }
I20260928 12:31:11.131137 14905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:11.131234 14810 master.cc:537] Master@127.8.13.61:34599 shutting down...
I20260928 12:31:11.131637 14884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:11.131642 14904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:11.131722 14884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } }
I20260928 12:31:11.131891 14884 leader_election.cc:288] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429)
I20260928 12:31:11.132685 14356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73a8920c213d4947a6504ea84885e8fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2726055b354a40239b82c2c722552b5f" is_pre_election: true
W20260928 12:31:11.132799 14331 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 12:31:11.133098 14828 leader_election.cc:302] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 73a8920c213d4947a6504ea84885e8fb; no voters: 2726055b354a40239b82c2c722552b5f
I20260928 12:31:11.133198 14884 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260928 12:31:11.133618 14331 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f -> Peer 73a8920c213d4947a6504ea84885e8fb (127.8.13.61:34599): Couldn't send request to peer 73a8920c213d4947a6504ea84885e8fb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 12:31:11.134647 14810 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:11.134725 14810 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:11.134755 14810 tablet_replica.cc:330] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: stopping tablet replica
I20260928 12:31:11.147927 14810 master.cc:552] Master@127.8.13.61:34599 shutdown complete.
I20260928 12:31:11.147994 14810 master_runner.cc:305] Clearing existing system tablet
I20260928 12:31:11.148126 14810 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:11.148172 14810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:11.148905 14810 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:31:11.149633 14909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.149868 14810 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:11.149930 14810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:31:11.149955 14810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/wal
uuid: "73a8920c213d4947a6504ea84885e8fb"
format_stamp: "Formatted at 2026-09-28 12:31:11 on dist-test-slave-g31s"
server_key: "bedead754dc66f7f35b6969207ed1c4b"
server_key_iv: "41fc9d988608c8ed743bc1e73809f3a7"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.150804 14810 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:11.152493 14810 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 12:31:11.152566 14810 log.cc:1196] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 12:31:11.152756 14810 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: Deleting consensus metadata
I20260928 12:31:11.152870 14810 master_runner.cc:315] Copying system tablet from 127.8.13.62:45429
I20260928 12:31:11.153509 14810 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: tablet copy: Beginning tablet copy session from remote peer at address 127.8.13.62:45429
I20260928 12:31:11.159929 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.153923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.8.13.62:45429)
0928 12:31:11.153978 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.153980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.154044 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:31:11.154129 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.154130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.154139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.154186 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.154572 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.155091 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.156189 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.156386 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.156451 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.157486 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.157487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.157487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.157493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.158806 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.159201 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.159202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.159203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.159263 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.159471 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.159472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.159472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.159593 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.159649 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.159797 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.159860 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:31:11.159997 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.153847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:48870)
0928 12:31:11.153974 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.153985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.154063 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.154203 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.154204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.154408 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.154496 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.154499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.155168 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.155170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.156078 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.156081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.156467 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.156468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.156535 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.156989 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.156993 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.157392 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.158881 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.158882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.158883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.159108 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.159327 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.159328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.159328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.159397 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.159650 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.159651 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.159700 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.159768 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.159880 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260928 12:31:11.160619 14366 tablet_copy_service.cc:140] P 2726055b354a40239b82c2c722552b5f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 73a8920c213d4947a6504ea84885e8fb ({username='oryx', principal='oryx/127.8.13.61@KRBTEST.COM'} at 127.0.0.1:48870)
I20260928 12:31:11.160708 14366 tablet_copy_service.cc:161] P 2726055b354a40239b82c2c722552b5f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 73a8920c213d4947a6504ea84885e8fb at {username='oryx', principal='oryx/127.8.13.61@KRBTEST.COM'} at 127.0.0.1:48870: session id = 73a8920c213d4947a6504ea84885e8fb-00000000000000000000000000000000
I20260928 12:31:11.161568 14366 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 12:31:11.162346 14810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 12:31:11.163425 14810 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: tablet copy: Starting download of 0 data blocks...
I20260928 12:31:11.163561 14810 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: tablet copy: Starting download of 1 WAL segments...
I20260928 12:31:11.164676 14810 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 12:31:11.165548 14366 tablet_copy_service.cc:342] P 2726055b354a40239b82c2c722552b5f: Request end of tablet copy session 73a8920c213d4947a6504ea84885e8fb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.13.61@KRBTEST.COM'} at 127.0.0.1:48870
I20260928 12:31:11.165601 14366 tablet_copy_service.cc:433] P 2726055b354a40239b82c2c722552b5f: ending tablet copy session 73a8920c213d4947a6504ea84885e8fb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 73a8920c213d4947a6504ea84885e8fb
I20260928 12:31:11.168156 14810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:11.169296 14917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.169324 14916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.169456 14919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:11.169660 14810 server_base.cc:1034] running on GCE node
I20260928 12:31:11.169749 14810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:11.169910 14810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:11.171039 14810 hybrid_clock.cc:648] HybridClock initialized: now 1790598671171007 us; error 50 us; skew 500 ppm
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:11.172844 14810 init.cc:377] Logged in from keytab as oryx/127.8.13.61@KRBTEST.COM (short username oryx)
I20260928 12:31:11.173148 14810 webserver.cc:460] Webserver started at http://127.8.13.61:46081/ using document root <none> and password file <none>
I20260928 12:31:11.173269 14810 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:11.173307 14810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:11.174049 14810 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:11.174594 14926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.174728 14810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:31:11.174772 14810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/wal
uuid: "73a8920c213d4947a6504ea84885e8fb"
format_stamp: "Formatted at 2026-09-28 12:31:11 on dist-test-slave-g31s"
server_key: "bedead754dc66f7f35b6969207ed1c4b"
server_key_iv: "41fc9d988608c8ed743bc1e73809f3a7"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.174837 14810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:11.190747 14810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:11.192916 14810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:11.195858 14810 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.61:34599
I20260928 12:31:11.196126 14978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.61:34599 every 8 connection(s)
I20260928 12:31:11.196213 14810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
I20260928 12:31:11.197023 14979 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:31:11.197446 14979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: Bootstrap starting.
I20260928 12:31:11.200026 14979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:31:11.200246 14979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: Bootstrap complete.
I20260928 12:31:11.200480 14979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } }
I20260928 12:31:11.200552 14979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73a8920c213d4947a6504ea84885e8fb, State: Initialized, Role: LEARNER
I20260928 12:31:11.200579 14979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } }
I20260928 12:31:11.200743 14979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:11.200764 14982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } }
I20260928 12:31:11.200879 14982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: This master's current role is: LEARNER
I20260928 12:31:11.202148 14993 catalog_manager.cc:1259] Loaded cluster ID: ded15d47e7974d518d3d52c27a1cda2a
I20260928 12:31:11.202194 14993 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: loading cluster ID for follower catalog manager: success
I20260928 12:31:11.203299 14993 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: acquiring CA information for follower catalog manager: success
I20260928 12:31:11.203567 14993 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:31:11.252197 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.245376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.8.13.61:34599)
0928 12:31:11.245533 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.245535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.245543 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:11.245568 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.245569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.245580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.245639 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.246226 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.246878 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.247610 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.247832 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.247893 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.249221 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.249223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.249223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.249232 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.251017 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.251391 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.251392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.251392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.251449 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.251609 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.251609 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.251610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.251728 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.251818 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.251935 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.252029 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.8.13.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.60.keytab.
Entry for principal oryx/127.8.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.60.keytab.
I20260928 12:31:11.281886 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.60:40687
--webserver_interface=127.8.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.60.keytab
--principal=oryx/127.8.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:45429,127.8.13.61:34599,127.8.13.60:40687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:11.405874 14997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:11.406149 14997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:11.406175 14997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:11.407560 14997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:11.407610 14997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:11.407624 14997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:11.407706 14997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:11.407778 14997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:11.409708 14997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:45429,127.8.13.61:34599,127.8.13.60:40687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.60.keytab
--principal=oryx/127.8.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.60:40687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:11.410039 14997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:11.410283 14997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:11.415573 15005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.415608 15003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.415835 15002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:11.416025 14997 server_base.cc:1034] running on GCE node
I20260928 12:31:11.416378 14997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:11.416690 14997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:11.417868 14997 hybrid_clock.cc:648] HybridClock initialized: now 1790598671417853 us; error 42 us; skew 500 ppm
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:11.420404 14997 init.cc:377] Logged in from keytab as oryx/127.8.13.60@KRBTEST.COM (short username oryx)
I20260928 12:31:11.420876 14997 webserver.cc:460] Webserver started at http://127.8.13.60:34785/ using document root <none> and password file <none>
I20260928 12:31:11.421099 14997 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:11.421147 14997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:11.421272 14997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:11.422401 14997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data/instance:
uuid: "b7fbc38338d64aa198c42bfdbb4698f4"
format_stamp: "Formatted at 2026-09-28 12:31:11 on dist-test-slave-g31s"
server_key: "294282afcd3e50ba4f270f94e7637ddd"
server_key_iv: "891704624a0dd43f0168c2a2488da99d"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.422741 14997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/wal/instance:
uuid: "b7fbc38338d64aa198c42bfdbb4698f4"
format_stamp: "Formatted at 2026-09-28 12:31:11 on dist-test-slave-g31s"
server_key: "294282afcd3e50ba4f270f94e7637ddd"
server_key_iv: "891704624a0dd43f0168c2a2488da99d"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.424273 14997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:11.425390 15012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.425658 14997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:11.425765 14997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/wal
uuid: "b7fbc38338d64aa198c42bfdbb4698f4"
format_stamp: "Formatted at 2026-09-28 12:31:11 on dist-test-slave-g31s"
server_key: "294282afcd3e50ba4f270f94e7637ddd"
server_key_iv: "891704624a0dd43f0168c2a2488da99d"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.425871 14997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:11.439230 14997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:11.441457 14997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:11.441607 14997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:11.449004 14997 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.60:40687
I20260928 12:31:11.449183 15064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.60:40687 every 8 connection(s)
I20260928 12:31:11.449568 14997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
I20260928 12:31:11.451512 15065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:11.454953 15065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:31:11.458662 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 14997
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.60@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.13.60@KRBTEST.COM
I20260928 12:31:11.467392 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.456103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.8.13.62:45429)
0928 12:31:11.456209 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.456212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.456675 (+ 463us) server_negotiation.cc:407] Connection header received
0928 12:31:11.457645 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.457647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.457663 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.457746 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.459227 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.459749 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.460981 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.461177 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.461234 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.464294 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.464296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.464297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.464306 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.465656 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.466364 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.466365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.466366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.466423 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.466707 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.466708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.466708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.466829 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.466890 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.467245 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.467315 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 12:31:11.467711 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.455963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:48872)
0928 12:31:11.456449 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.456501 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.457503 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.457872 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.457878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.458456 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.459094 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.459105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.459879 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.459880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.460871 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.460874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.461258 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.461259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.461451 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.462279 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.462292 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.464186 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.465751 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.465762 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.465765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.466269 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.466496 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.466497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.466497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.466636 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.466903 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.466908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.467091 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.467204 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.467392 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":266,"threads_started":1}
I20260928 12:31:11.468310 15065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:31:11.468811 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.459240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.8.13.60:40687)
0928 12:31:11.459644 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.459649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.459664 (+ 15us) server_negotiation.cc:407] Connection header received
0928 12:31:11.459687 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.459689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.459728 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.459800 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.460375 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.461456 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.462358 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.462764 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.462828 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.465234 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.465246 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.465246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.465253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.467525 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.467915 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.467918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.467918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.467983 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.468180 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.468181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.468181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.468326 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.468468 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.468556 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.468636 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":265,"threads_started":1}
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.60@KRBTEST.COM for oryx/127.8.13.61@KRBTEST.COM
I20260928 12:31:11.475999 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.468635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.8.13.61:34599)
0928 12:31:11.468677 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.468679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.468956 (+ 277us) server_negotiation.cc:407] Connection header received
0928 12:31:11.468986 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.468987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.469001 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.469052 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.469704 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.470272 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.470922 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.471140 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.471195 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.473464 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.473466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.473466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.473474 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.474920 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.475313 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.475313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.475314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.475365 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.475581 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.475582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.475582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.475692 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.475737 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.475857 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.475930 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:31:11.476022 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.468572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:34599 (local address 127.0.0.1:57440)
0928 12:31:11.468887 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.468896 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.468943 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.469353 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.469353 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.469476 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.469619 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.469621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.470344 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.470345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.470850 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.470852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.471200 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.471201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.471337 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.471755 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.471759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.473342 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.475035 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.475037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.475038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.475236 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.475463 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.475463 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.475464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.475527 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.475734 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.475735 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.475778 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.475838 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.475951 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303}
I20260928 12:31:11.476603 15065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.60@KRBTEST.COM for oryx/127.8.13.60@KRBTEST.COM
I20260928 12:31:11.484076 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.477011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:40687 (local address 127.0.0.1:46490)
0928 12:31:11.477084 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.477099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.477182 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.477306 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.477307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.477467 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.477532 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.477534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.478246 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.478247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.478757 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.478760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.479314 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.479315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.479364 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.479825 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.479829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.481288 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.482965 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.482966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.482967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.483134 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.483382 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.483383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.483384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.483460 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.483751 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.483752 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.483797 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.483873 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.483976 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 12:31:11.484129 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.477075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46490 (local address 127.8.13.60:40687)
0928 12:31:11.477118 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.477120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.477163 (+ 43us) server_negotiation.cc:407] Connection header received
0928 12:31:11.477232 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.477233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.477243 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.477290 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.477592 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.478136 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.478855 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.479191 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.479259 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.481406 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.481407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.481408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.481415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.482874 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.483233 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.483234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.483235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.483291 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.483516 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.483517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.483517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.483661 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.483736 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.483908 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.484026 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":18}
I20260928 12:31:11.487119 15065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: Bootstrap starting.
I20260928 12:31:11.488696 15065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:11.489423 15065 log.cc:824] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:11.490569 15065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: No bootstrap required, opened a new log
I20260928 12:31:11.492624 15065 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } }
I20260928 12:31:11.492795 15065 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:11.492818 15065 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fbc38338d64aa198c42bfdbb4698f4, State: Initialized, Role: FOLLOWER
I20260928 12:31:11.492920 15065 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } }
I20260928 12:31:11.493244 15065 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:11.493379 15070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } } }
I20260928 12:31:11.493469 15070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:31:11.496735 15081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:31:11.496966 15081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:31:11.500515 14997 master_runner.cc:428] Detected that this master b7fbc38338d64aa198c42bfdbb4698f4 is joining an existing cluster
I20260928 12:31:11.500582 14997 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.13.60:40687
I20260928 12:31:11.508379 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.501889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:34599 (local address 127.0.0.1:57448)
0928 12:31:11.502135 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.502151 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.502242 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.502460 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.502461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.502671 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.502741 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.502743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.503399 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.503400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.504120 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.504123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.504469 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.504470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.504546 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.505055 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.505060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.505382 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.507230 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.507231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.507231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.507477 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.507666 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.507666 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.507667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.507725 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.507982 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.507983 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.508025 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.508128 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.508237 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":98,"threads_started":1}
I20260928 12:31:11.508792 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.501929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.8.13.61:34599)
0928 12:31:11.501989 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.501992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.502235 (+ 243us) server_negotiation.cc:407] Connection header received
0928 12:31:11.502320 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.502321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.502333 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.502389 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.502820 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.503317 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.504201 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.504395 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.504453 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.505498 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.505499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.505499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.505507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.507140 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.507546 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.507548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.507548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.507606 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.507782 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.507783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.507783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.507903 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.507958 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.508616 (+ 658us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.508701 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:31:11.512670 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.506043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.8.13.62:45429)
0928 12:31:11.506100 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.506103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.506278 (+ 175us) server_negotiation.cc:407] Connection header received
0928 12:31:11.506621 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.506622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.506635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.506666 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.507490 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.508185 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.509215 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.509482 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.509520 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.510422 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.510423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.510424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.510431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.511621 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.511980 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.511981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.511981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.512037 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.512190 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.512191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.512191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.512302 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.512358 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.512500 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.512574 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:31:11.512684 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.506073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:48886)
0928 12:31:11.506223 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.506233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.506535 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.506741 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.506742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.507009 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.507398 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.507400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.508265 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.508266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.509135 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.509137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.509582 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.509582 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.509632 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.510055 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.510060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.510333 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.511697 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.511698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.511698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.511907 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.512085 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.512085 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.512086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.512141 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.512355 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.512356 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.512412 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.512473 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.512575 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598670, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.62@KRBTEST.COM for oryx/127.8.13.60@KRBTEST.COM
I20260928 12:31:11.520059 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 14408
I20260928 12:31:11.524806 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:39495
--local_ip_for_outbound_sockets=127.8.13.1
--tserver_master_addrs=127.8.13.62:45429
--webserver_port=42007
--webserver_interface=127.8.13.1
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab
--principal=oryx/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.13.62:45429,127.8.13.61:34599
--tserver_master_addrs=127.8.13.62:45429,127.8.13.61:34599,127.8.13.60:40687 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:31:11.528720 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.514952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:40687 (local address 127.0.0.1:46506)
0928 12:31:11.515008 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.515022 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.515075 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.515416 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.515417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.515554 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.515600 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.515601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.516664 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.516665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.517189 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.517191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.517694 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.517694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.517741 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.518151 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.518156 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.519565 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.527515 (+ 7950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.527516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.527517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.527748 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.527983 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.527984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.527984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.528065 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.528378 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.528379 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.528446 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.528494 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.528609 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 12:31:11.529117 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.514977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.8.13.60:40687)
0928 12:31:11.515195 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.515198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.515211 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:31:11.515269 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.515270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.515293 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.515383 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.515727 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.516555 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.517294 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.517624 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.517687 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.519658 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.519659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.519659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.519667 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.527405 (+ 7738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.527848 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.527849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.527850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.527905 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.528151 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.528152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.528152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.528297 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.528902 (+ 605us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.528939 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.529013 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":120,"threads_started":1}
I20260928 12:31:11.529708 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.506381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:40687 (local address 127.0.0.1:46496)
0928 12:31:11.506606 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.506616 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.506746 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.506898 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.506899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.507144 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.507208 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.507210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.508094 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.508095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.509293 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.509296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.509758 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.509759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.509844 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.526456 (+ 16612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.526462 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.527063 (+ 601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.528801 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.528802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.528802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.528978 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.529138 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.529139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.529139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.529184 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.529419 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.529421 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.529468 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.529524 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.529621 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260928 12:31:11.529732 14346 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.13.60:40687: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4"
member_type: NON_VOTER
last_known_addr {
host: "127.8.13.60"
port: 40687
}
attrs {
promote: true
}
}
dest_uuid: "2726055b354a40239b82c2c722552b5f"
cas_config_opid_index: 5
I20260928 12:31:11.529965 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.506335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.8.13.60:40687)
0928 12:31:11.506385 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.506386 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.506672 (+ 286us) server_negotiation.cc:407] Connection header received
0928 12:31:11.506814 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.506815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.506823 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.506848 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.507287 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.508011 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.509379 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.509662 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.509696 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.527195 (+ 17499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.527197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.527197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.527205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.528730 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.529040 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.529041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.529041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.529087 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.529237 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.529238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.529238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.529359 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.529417 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.529835 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.529901 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:31:11.529992 14346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } }
I20260928 12:31:11.530385 14905 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b7fbc38338d64aa198c42bfdbb4698f4 (127.8.13.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.530649 15039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:11.531286 14905 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.13.60:40687
I20260928 12:31:11.531324 15039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 FOLLOWER]: Refusing update from remote peer 2726055b354a40239b82c2c722552b5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 12:31:11.531355 14905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.531415 14905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:11.531605 14904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:11.532838 15039 raft_consensus.cc:858] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } }
W20260928 12:31:11.532920 15039 raft_consensus.cc:1565] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } }
I20260928 12:31:11.533558 15039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 73a8920c213d4947a6504ea84885e8fb (127.8.13.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } }
I20260928 12:31:11.535899 15070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2726055b354a40239b82c2c722552b5f. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } }
I20260928 12:31:11.535991 15070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 12:31:11.536028 15070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } }
I20260928 12:31:11.536062 15070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 12:31:11.536084 15070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } }
I20260928 12:31:11.536116 15070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 12:31:11.536933 14997 master.cc:537] Master@127.8.13.60:40687 shutting down...
I20260928 12:31:11.538216 14904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.538335 14904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: This master's current role is: LEADER
W20260928 12:31:11.538930 14332 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f -> Peer b7fbc38338d64aa198c42bfdbb4698f4 (127.8.13.60:40687): Couldn't send request to peer b7fbc38338d64aa198c42bfdbb4698f4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 12:31:11.539458 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.530506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.8.13.61:34599)
0928 12:31:11.530549 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.530551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.531850 (+ 1299us) server_negotiation.cc:407] Connection header received
0928 12:31:11.531878 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.531879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.531906 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.531965 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:11.534202 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.534694 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.536167 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.536623 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.536690 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.536759 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.539325 (+ 2566us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.539386 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:31:11.540062 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.530413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:34599 (local address 127.0.0.1:57452)
0928 12:31:11.531497 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.531508 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.531551 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.534024 (+ 2473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.534026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.534028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:11.534084 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.534086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.534769 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.534770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.535935 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.535939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.536700 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.536701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.536765 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.536787 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.536849 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.536900 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071}
I20260928 12:31:11.540306 14997 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260928 12:31:11.540323 14953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 LEARNER]: Refusing update from remote peer 2726055b354a40239b82c2c722552b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 12:31:11.540416 14997 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 12:31:11.540450 14997 tablet_replica.cc:330] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: stopping tablet replica
I20260928 12:31:11.540680 14904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 12:31:11.541249 14953 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b7fbc38338d64aa198c42bfdbb4698f4 (127.8.13.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.542475 14982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2726055b354a40239b82c2c722552b5f. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.542680 14904 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f: attempting to promote NON_VOTER 73a8920c213d4947a6504ea84885e8fb to VOTER
I20260928 12:31:11.542898 14904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } }
I20260928 12:31:11.542568 14982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: This master's current role is: LEARNER
I20260928 12:31:11.542510 15101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: NON_VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: true } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.543444 15101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: This master's current role is: LEARNER
W20260928 12:31:11.543898 14332 connection.cc:535] client connection to 127.8.13.60:40687 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 12:31:11.543954 14332 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f -> Peer b7fbc38338d64aa198c42bfdbb4698f4 (127.8.13.60:40687): Couldn't send request to peer b7fbc38338d64aa198c42bfdbb4698f4. Status: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260928 12:31:11.544333 14953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 LEARNER]: Refusing update from remote peer 2726055b354a40239b82c2c722552b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 12:31:11.544526 14905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 12:31:11.545197 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.545038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:40687 (local address 127.0.0.1:46516)
0928 12:31:11.545121 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.545159 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 12:31:11.545501 14904 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 73a8920c213d4947a6504ea84885e8fb (127.8.13.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.546378 14904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.546456 14904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:11.546504 14904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.546547 14904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:11.546780 14953 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 73a8920c213d4947a6504ea84885e8fb (127.8.13.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.547613 15101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.547698 15101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:31:11.556242 14997 master.cc:552] Master@127.8.13.60:40687 shutdown complete.
I20260928 12:31:11.556303 14997 master_runner.cc:305] Clearing existing system tablet
I20260928 12:31:11.556443 14997 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:11.556484 14997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:11.557102 14997 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260928 12:31:11.557592 15103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.557686 14997 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:11.557726 14997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.557749 14997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/wal
uuid: "b7fbc38338d64aa198c42bfdbb4698f4"
format_stamp: "Formatted at 2026-09-28 12:31:11 on dist-test-slave-g31s"
server_key: "294282afcd3e50ba4f270f94e7637ddd"
server_key_iv: "891704624a0dd43f0168c2a2488da99d"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.558506 14997 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:11.560505 14997 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 12:31:11.560587 14997 log.cc:1196] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 12:31:11.560806 14997 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: Deleting consensus metadata
I20260928 12:31:11.560930 14997 master_runner.cc:315] Copying system tablet from 127.8.13.62:45429
I20260928 12:31:11.561513 14997 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: tablet copy: Beginning tablet copy session from remote peer at address 127.8.13.62:45429
I20260928 12:31:11.567698 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.561978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48902 (local address 127.8.13.62:45429)
0928 12:31:11.562030 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.562032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.562130 (+ 98us) server_negotiation.cc:407] Connection header received
0928 12:31:11.562198 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.562199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.562210 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.562256 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.562759 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.563267 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.564090 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.564288 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.564346 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.565310 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.565311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.565312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.565320 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.566633 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.566964 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.566966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.566966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.567026 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.567187 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.567189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.567189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.567310 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.567368 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.567507 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.567594 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:31:11.567703 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.561930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:48902)
0928 12:31:11.562061 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.562070 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.562135 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.562261 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.562261 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.562440 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.562681 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.562683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.563347 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.563347 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.564009 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.564012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.564355 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.564356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.564411 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.564895 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.564900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.565219 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.566720 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.566721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.566722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.566902 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.567079 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.567079 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.567080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.567140 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.567366 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.567367 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.567418 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.567478 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.567596 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260928 12:31:11.568382 14366 tablet_copy_service.cc:140] P 2726055b354a40239b82c2c722552b5f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b7fbc38338d64aa198c42bfdbb4698f4 ({username='oryx', principal='oryx/127.8.13.60@KRBTEST.COM'} at 127.0.0.1:48902)
I20260928 12:31:11.568445 14366 tablet_copy_service.cc:161] P 2726055b354a40239b82c2c722552b5f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b7fbc38338d64aa198c42bfdbb4698f4 at {username='oryx', principal='oryx/127.8.13.60@KRBTEST.COM'} at 127.0.0.1:48902: session id = b7fbc38338d64aa198c42bfdbb4698f4-00000000000000000000000000000000
I20260928 12:31:11.568930 14366 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 12:31:11.569443 14997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 12:31:11.570621 14997 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: tablet copy: Starting download of 0 data blocks...
I20260928 12:31:11.570767 14997 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: tablet copy: Starting download of 1 WAL segments...
I20260928 12:31:11.571477 14997 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 12:31:11.572304 14366 tablet_copy_service.cc:342] P 2726055b354a40239b82c2c722552b5f: Request end of tablet copy session b7fbc38338d64aa198c42bfdbb4698f4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.13.60@KRBTEST.COM'} at 127.0.0.1:48902
I20260928 12:31:11.572402 14366 tablet_copy_service.cc:433] P 2726055b354a40239b82c2c722552b5f: ending tablet copy session b7fbc38338d64aa198c42bfdbb4698f4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b7fbc38338d64aa198c42bfdbb4698f4
I20260928 12:31:11.575131 14997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:11.576375 15113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:11.576573 14997 server_base.cc:1034] running on GCE node
W20260928 12:31:11.576579 15110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.577955 15111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:11.578131 14997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:11.578301 14997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:11.579433 14997 hybrid_clock.cc:648] HybridClock initialized: now 1790598671579427 us; error 27 us; skew 500 ppm
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:11.580727 14997 init.cc:377] Logged in from keytab as oryx/127.8.13.60@KRBTEST.COM (short username oryx)
I20260928 12:31:11.580991 14997 webserver.cc:460] Webserver started at http://127.8.13.60:44525/ using document root <none> and password file <none>
I20260928 12:31:11.581105 14997 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:11.581139 14997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:11.581876 14997 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.582425 15120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.582535 14997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:31:11.582577 14997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/wal
uuid: "b7fbc38338d64aa198c42bfdbb4698f4"
format_stamp: "Formatted at 2026-09-28 12:31:11 on dist-test-slave-g31s"
server_key: "294282afcd3e50ba4f270f94e7637ddd"
server_key_iv: "891704624a0dd43f0168c2a2488da99d"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.582643 14997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:11.597631 14997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:11.599545 14997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:11.602598 14997 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.60:40687
I20260928 12:31:11.602938 14997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
I20260928 12:31:11.605989 15172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.60:40687 every 8 connection(s)
I20260928 12:31:11.606638 15173 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:31:11.607013 15173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: Bootstrap starting.
I20260928 12:31:11.608770 15173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:31:11.608971 15173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: Bootstrap complete.
I20260928 12:31:11.609169 15173 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } }
I20260928 12:31:11.609242 15173 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7fbc38338d64aa198c42bfdbb4698f4, State: Initialized, Role: LEARNER
I20260928 12:31:11.609289 15173 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } }
I20260928 12:31:11.609478 15173 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:11.609481 15176 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.609683 15176 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: This master's current role is: LEARNER
I20260928 12:31:11.610961 15187 catalog_manager.cc:1259] Loaded cluster ID: ded15d47e7974d518d3d52c27a1cda2a
I20260928 12:31:11.611049 15187 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: loading cluster ID for follower catalog manager: success
I20260928 12:31:11.612125 15187 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: acquiring CA information for follower catalog manager: success
I20260928 12:31:11.612424 15187 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 12:31:11.655270 15091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:11.655740 15091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:11.655850 15091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:11.658289 15091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:11.658377 15091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:11.658425 15091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:11.660585 15091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab
--principal=oryx/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:39495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=42007
--tserver_master_addrs=127.8.13.62:45429,127.8.13.61:34599,127.8.13.60:40687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:11.660913 15091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:11.661262 15091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:11.667304 15192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.667315 15195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.667322 15193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:11.667485 15091 server_base.cc:1034] running on GCE node
I20260928 12:31:11.667776 15091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:11.668045 15091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:11.669256 15091 hybrid_clock.cc:648] HybridClock initialized: now 1790598671669229 us; error 43 us; skew 500 ppm
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:11.671908 15091 init.cc:377] Logged in from keytab as oryx/127.8.13.1@KRBTEST.COM (short username oryx)
I20260928 12:31:11.672422 15091 webserver.cc:460] Webserver started at http://127.8.13.1:42007/ using document root <none> and password file <none>
I20260928 12:31:11.672641 15091 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:11.672816 15091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:11.674906 15091 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:11.675571 15202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.675776 15091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:11.675895 15091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "b8acc77419c94659863723a919a041f3"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "74941ab07bd18e5c848afc13c9682d66"
server_key_iv: "25e6ae7d112f1f9923d3e42b132654c7"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.676283 15091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:11.684229 15091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:11.686503 15091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:11.686659 15091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:11.686890 15091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:11.687201 15091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:11.687234 15091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.687266 15091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:11.687289 15091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.694281 15091 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:39495
I20260928 12:31:11.694344 15315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:39495 every 8 connection(s)
I20260928 12:31:11.694759 15091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:31:11.701503 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 15091
I20260928 12:31:11.701709 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 14542
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.61@KRBTEST.COM
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.60@KRBTEST.COM
I20260928 12:31:11.709052 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:44661
--local_ip_for_outbound_sockets=127.8.13.2
--tserver_master_addrs=127.8.13.62:45429
--webserver_port=41733
--webserver_interface=127.8.13.2
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab
--principal=oryx/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.13.62:45429,127.8.13.61:34599
--tserver_master_addrs=127.8.13.62:45429,127.8.13.61:34599,127.8.13.60:40687 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:31:11.711184 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.696232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:40687 (local address 127.8.13.1:50425)
0928 12:31:11.696649 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.696659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.697190 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.697443 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.697444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.698073 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.698508 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.698514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.699892 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.699893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.701349 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.701355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.702368 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.702370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.702443 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.703420 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.703425 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.706599 (+ 3174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.708907 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.708922 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.708926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.709434 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.709731 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.709732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.709732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.709932 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.710330 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.710331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.710542 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.710755 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.710976 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260928 12:31:11.711378 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.695907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:60257 (local address 127.8.13.61:34599)
0928 12:31:11.695977 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.695979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.696223 (+ 244us) server_negotiation.cc:407] Connection header received
0928 12:31:11.697246 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.697247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.697262 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.697323 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.698762 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.699516 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.701829 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.702122 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.702176 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.706719 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.706722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.706722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.706733 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.708845 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.709529 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.709530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.709531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.709670 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.710705 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.710707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.710707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.710897 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.710942 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.711193 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.711287 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:31:11.711417 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.695849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:34599 (local address 127.8.13.1:60257)
0928 12:31:11.696049 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.696122 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.697107 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.697423 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.697429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.698393 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.698611 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.698618 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.699706 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.699706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.701621 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.701624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.702259 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.702259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.702420 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.703331 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.703335 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.706332 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.708924 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.708924 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.708927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.709434 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.709731 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.709732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.709733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.709934 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.711010 (+ 1076us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.711010 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.711053 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.711111 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.711224 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":128,"thread_start_us":102,"threads_started":1}
I20260928 12:31:11.711527 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.696304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:50425 (local address 127.8.13.60:40687)
0928 12:31:11.696506 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.696508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.696717 (+ 209us) server_negotiation.cc:407] Connection header received
0928 12:31:11.697300 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.697301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.697320 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.697426 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.698607 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.699798 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.701505 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.702214 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.702291 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.707006 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.707008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.707008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.707016 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.708798 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.709578 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.709579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.709580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.709647 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.710006 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.710007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.710008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.710227 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.710684 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.711343 (+ 659us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.711442 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260928 12:31:11.711797 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.696541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:50311 (local address 127.8.13.62:45429)
0928 12:31:11.696646 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.696649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.696851 (+ 202us) server_negotiation.cc:407] Connection header received
0928 12:31:11.697387 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.697388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.697398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.697443 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.698814 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.699387 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.701596 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.701924 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.701981 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.706065 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.706069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.706069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.706081 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.708148 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.709530 (+ 1382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.709531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.709531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.709607 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.710094 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.710095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.710095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.710223 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.710278 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.711593 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.711696 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 12:31:11.711875 15318 heartbeater.cc:347] Connected to a master server at 127.8.13.61:34599
I20260928 12:31:11.711937 15316 heartbeater.cc:347] Connected to a master server at 127.8.13.60:40687
I20260928 12:31:11.711982 15316 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:11.711982 15318 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:11.712194 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.696419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.1:50311)
0928 12:31:11.696767 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.696778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.697130 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.697496 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.697497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.698393 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.698677 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.698678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.699687 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.699694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.701448 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.701452 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.702131 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.702133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.702420 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.703133 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.703144 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.705888 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.708908 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.708922 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.708926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.709434 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.709750 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.709751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.709751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.710029 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.710320 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.710328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.710542 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.711910 (+ 1368us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.712018 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":153,"thread_start_us":149,"threads_started":1}
I20260928 12:31:11.712200 15318 heartbeater.cc:511] Master 127.8.13.61:34599 requested a full tablet report, sending...
I20260928 12:31:11.712505 15316 heartbeater.cc:511] Master 127.8.13.60:40687 requested a full tablet report, sending...
I20260928 12:31:11.712777 14943 ts_manager.cc:194] Registered new tserver with Master: b8acc77419c94659863723a919a041f3 (127.8.13.1:39495)
I20260928 12:31:11.712829 15137 ts_manager.cc:194] Registered new tserver with Master: b8acc77419c94659863723a919a041f3 (127.8.13.1:39495)
I20260928 12:31:11.713347 15317 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45429
I20260928 12:31:11.713392 15317 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:11.713454 15317 heartbeater.cc:511] Master 127.8.13.62:45429 requested a full tablet report, sending...
I20260928 12:31:11.713719 14346 ts_manager.cc:194] Re-registered known tserver with Master: b8acc77419c94659863723a919a041f3 (127.8.13.1:39495)
I20260928 12:31:11.714128 14346 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:50311
W20260928 12:31:11.796753 15324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:11.797021 15324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:11.797050 15324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:11.798470 15324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:11.798537 15324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:11.798571 15324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:11.800335 15324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab
--principal=oryx/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:44661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=41733
--tserver_master_addrs=127.8.13.62:45429,127.8.13.61:34599,127.8.13.60:40687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:11.800617 15324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:11.800859 15324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:11.806802 15330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.806712 15332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.806712 15329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:11.807116 15324 server_base.cc:1034] running on GCE node
I20260928 12:31:11.807514 15324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:11.807793 15324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:11.808965 15324 hybrid_clock.cc:648] HybridClock initialized: now 1790598671808943 us; error 39 us; skew 500 ppm
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:11.811830 15324 init.cc:377] Logged in from keytab as oryx/127.8.13.2@KRBTEST.COM (short username oryx)
I20260928 12:31:11.812289 15324 webserver.cc:460] Webserver started at http://127.8.13.2:41733/ using document root <none> and password file <none>
I20260928 12:31:11.812533 15324 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:11.812601 15324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:11.814086 15324 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:11.814736 15339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.814880 15324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:31:11.814961 15324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "c33e75f6c55a4f39ae62d0547553141f"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "acf815b787770a1177d9579c6abd96d1"
server_key_iv: "a5428a417cfad92d3426080c9f55309b"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.815642 15324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:11.822971 15324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:11.825181 15324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:11.825299 15324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:11.825519 15324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:11.825790 15324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:11.825866 15324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.825892 15324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:11.825908 15324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.832808 15324 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:44661
I20260928 12:31:11.832859 15452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:44661 every 8 connection(s)
I20260928 12:31:11.833331 15324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:31:11.836911 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 15324
I20260928 12:31:11.837090 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 14676
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.61@KRBTEST.COM
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.60@KRBTEST.COM
I20260928 12:31:11.844758 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:46869
--local_ip_for_outbound_sockets=127.8.13.3
--tserver_master_addrs=127.8.13.62:45429
--webserver_port=42995
--webserver_interface=127.8.13.3
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab
--principal=oryx/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.13.62:45429,127.8.13.61:34599
--tserver_master_addrs=127.8.13.62:45429,127.8.13.61:34599,127.8.13.60:40687 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:31:11.847782 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.834294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:41741 (local address 127.8.13.61:34599)
0928 12:31:11.834378 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.834380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.834629 (+ 249us) server_negotiation.cc:407] Connection header received
0928 12:31:11.836020 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.836022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.836036 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.836087 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.837095 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.837882 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.840035 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.840354 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.840419 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.844386 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.844387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.844388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.844396 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.846155 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.846638 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.846639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.846640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.846700 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.846963 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.846964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.846964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.847104 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.847147 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.847591 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.847685 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 12:31:11.847779 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.834241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:32931 (local address 127.8.13.60:40687)
0928 12:31:11.834371 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.834373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.834630 (+ 257us) server_negotiation.cc:407] Connection header received
0928 12:31:11.835937 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.835938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.835958 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.836008 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.837122 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.837863 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.840043 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.840357 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.840410 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.844374 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.844375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.844376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.844384 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.845926 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.846619 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.846620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.846620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.846687 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.846958 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.846959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.846959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.847102 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.847146 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.847583 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.847688 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 12:31:11.847831 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.834616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:45763 (local address 127.8.13.62:45429)
0928 12:31:11.834658 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.834660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.834732 (+ 72us) server_negotiation.cc:407] Connection header received
0928 12:31:11.836155 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.836157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.836171 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.836227 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:11.837042 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.837884 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.840036 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.840354 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.840400 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.844022 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:11.844025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:11.844025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:11.844037 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:11.846455 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.846936 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.846938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.846939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.846995 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:11.847178 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:11.847179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:11.847180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:11.847302 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:11.847343 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.847678 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.847754 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:31:11.849257 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.834219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:40687 (local address 127.8.13.2:32931)
0928 12:31:11.834561 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.834573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.835817 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.836201 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.836202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.836593 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.837009 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.837012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.838042 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.838042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.839928 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.839932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.840514 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.840515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.840688 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.841384 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.841403 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.844272 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.846039 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.846056 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.846060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.846525 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.846751 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.846752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.846752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.846891 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.847194 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.847200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.847442 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.847544 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.847727 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":122,"thread_start_us":87,"threads_started":1}
I20260928 12:31:11.849273 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.834219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:34599 (local address 127.8.13.2:41741)
0928 12:31:11.834500 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.834545 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.835794 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.836115 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.836120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.836592 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.836926 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.836932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.838022 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.838022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.839889 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.839894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.840511 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.840512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.840686 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.841411 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.841416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.844269 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.846255 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.846256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.846257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.846557 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.846773 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.846773 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.846773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.846892 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.847190 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.847301 (+ 111us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.847446 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.848329 (+ 883us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.848413 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":101,"thread_start_us":96,"threads_started":1}
I20260928 12:31:11.849360 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.834534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.2:45763)
0928 12:31:11.834651 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.834658 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.835824 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.836246 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.836246 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.836601 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:11.836925 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.836932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.838022 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.838022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.839889 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.839894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.840513 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.840513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.840699 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.841630 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:11.841635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:11.843818 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:11.846551 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.846552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.846553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.846841 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.847061 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:11.847062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:11.847062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:11.847111 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:11.847387 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:11.847388 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:11.847446 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.848000 (+ 554us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.848128 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":43,"thread_start_us":67,"threads_started":1}
I20260928 12:31:11.849957 15453 heartbeater.cc:347] Connected to a master server at 127.8.13.60:40687
I20260928 12:31:11.850061 15453 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:11.850267 15453 heartbeater.cc:511] Master 127.8.13.60:40687 requested a full tablet report, sending...
I20260928 12:31:11.850754 15137 ts_manager.cc:194] Registered new tserver with Master: c33e75f6c55a4f39ae62d0547553141f (127.8.13.2:44661)
I20260928 12:31:11.851032 15455 heartbeater.cc:347] Connected to a master server at 127.8.13.61:34599
I20260928 12:31:11.851047 15454 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45429
I20260928 12:31:11.851130 15454 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:11.851214 15454 heartbeater.cc:511] Master 127.8.13.62:45429 requested a full tablet report, sending...
I20260928 12:31:11.851086 15455 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:11.851334 15455 heartbeater.cc:511] Master 127.8.13.61:34599 requested a full tablet report, sending...
I20260928 12:31:11.851464 14346 ts_manager.cc:194] Re-registered known tserver with Master: c33e75f6c55a4f39ae62d0547553141f (127.8.13.2:44661)
I20260928 12:31:11.851650 14943 ts_manager.cc:194] Registered new tserver with Master: c33e75f6c55a4f39ae62d0547553141f (127.8.13.2:44661)
I20260928 12:31:11.851930 14346 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:45763
W20260928 12:31:11.941674 15460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:11.942004 15460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:11.942036 15460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:11.943539 15460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:11.943598 15460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:11.943632 15460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:11.945402 15460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab
--principal=oryx/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:46869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=42995
--tserver_master_addrs=127.8.13.62:45429,127.8.13.61:34599,127.8.13.60:40687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:11.945706 15460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:11.945990 15460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:11.951149 15465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:11.951426 15460 server_base.cc:1034] running on GCE node
W20260928 12:31:11.951558 15468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:11.951146 15466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:11.951890 15460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:11.952256 15460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:11.953471 15460 hybrid_clock.cc:648] HybridClock initialized: now 1790598671953412 us; error 61 us; skew 500 ppm
I20260928 12:31:11.957026 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.954152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:40687 (local address 127.0.0.1:46526)
0928 12:31:11.954203 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.954214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.954259 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:11.954591 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:11.954592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:11.954595 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:11.954655 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.954658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.955441 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.955442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.956166 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:11.956168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.956760 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.956761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:11.956826 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.956830 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:11.956882 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:31:11.956944 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:31:11.957024 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.954215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.8.13.60:40687)
0928 12:31:11.954309 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.954312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.954330 (+ 18us) server_negotiation.cc:407] Connection header received
0928 12:31:11.954381 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.954383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.954425 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.954546 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:11.954802 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.955354 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.956255 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:11.956677 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:11.956744 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:11.956817 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:11.956899 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 12:31:11.956944 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 12:31:11 dist-test-slave-g31s krb5kdc[14301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:11.957371 15147 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 12:31:11.957501 15176 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2726055b354a40239b82c2c722552b5f. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:11.957597 15176 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: This master's current role is: LEARNER
I20260928 12:31:11.957926 15460 init.cc:377] Logged in from keytab as oryx/127.8.13.3@KRBTEST.COM (short username oryx)
I20260928 12:31:11.958663 15460 webserver.cc:460] Webserver started at http://127.8.13.3:42995/ using document root <none> and password file <none>
I20260928 12:31:11.958961 15460 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:11.959048 15460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:11.961058 15460 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:11.961769 15475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.962124 15460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:11.962239 15460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "8b411c034e084e3a9e811cb2aceea760"
format_stamp: "Formatted at 2026-09-28 12:31:10 on dist-test-slave-g31s"
server_key: "2cbc493d61df365cda8c85e002236180"
server_key_iv: "ff0416158305259a36c1ed31606fd3a4"
server_key_version: "encryptionkey@0"
I20260928 12:31:11.962765 15460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:11.980238 15460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:11.982591 15460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:11.982748 15460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:11.983032 15460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:11.983366 15460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:11.983405 15460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.983438 15460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:11.983461 15460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:11.996596 15460 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:46869
I20260928 12:31:11.996642 15588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:46869 every 8 connection(s)
I20260928 12:31:11.997076 15460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:31:12.001488 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 15460
Sep 28 12:31:12 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
Sep 28 12:31:12 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.61@KRBTEST.COM
Sep 28 12:31:12 dist-test-slave-g31s krb5kdc[14301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598671, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.60@KRBTEST.COM
I20260928 12:31:12.012558 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.997963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:48301 (local address 127.8.13.61:34599)
0928 12:31:11.998015 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.998018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.998275 (+ 257us) server_negotiation.cc:407] Connection header received
0928 12:31:11.999937 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.999939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.999952 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.000002 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:12.001453 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.002568 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.003952 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.004216 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.004273 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.009597 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:12.009599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:12.009600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:12.009608 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:12.011169 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.011482 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.011483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.011483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.011537 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.011696 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.011697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.011697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.011815 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:12.011871 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.012396 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.012465 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:31:12.012668 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.997966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:34599 (local address 127.8.13.3:48301)
0928 12:31:11.998142 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.998188 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.999573 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:12.000048 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:12.000049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:12.000876 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:12.001301 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.001309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.002908 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.002909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.003854 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.003857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.004283 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.004284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.004347 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.005725 (+ 1378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:12.005730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:12.009502 (+ 3772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:12.011252 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:12.011253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:12.011254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:12.011414 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:12.011590 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:12.011591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:12.011591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:12.011646 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:12.011905 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:12.011912 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:12.012095 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 12:31:12.012298 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 12:31:12.012465 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":93,"thread_start_us":99,"threads_started":1}
I20260928 12:31:12.012895 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.998253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.3:44185)
0928 12:31:11.998455 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.998465 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.999592 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:12.000098 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:12.000099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:12.000874 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:12.001319 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.001325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.002525 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.002527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.003700 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.003703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.004151 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.004153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.004302 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.005672 (+ 1370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:12.005685 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:12.009096 (+ 3411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:12.010956 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:12.010965 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:12.010969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:12.011277 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:12.011495 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:12.011496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:12.011496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:12.011632 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:12.011917 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:12.011918 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:12.012097 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 12:31:12.012210 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 12:31:12.012409 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":46,"thread_start_us":57,"threads_started":1}
I20260928 12:31:12.013180 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.998331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:44185 (local address 127.8.13.62:45429)
0928 12:31:11.998387 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.998389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.998522 (+ 133us) server_negotiation.cc:407] Connection header received
0928 12:31:11.999789 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.999791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.999802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.999831 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:12.001459 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.002246 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.003799 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.004061 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.004127 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.009182 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:12.009183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:12.009184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:12.009191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:12.010853 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.011359 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.011360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.011361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.011431 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.011699 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.011700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.011700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.011827 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:12.011892 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.013014 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.013094 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:31:12.013207 15591 heartbeater.cc:347] Connected to a master server at 127.8.13.61:34599
I20260928 12:31:12.013300 15591 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:12.013422 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.998102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:58585 (local address 127.8.13.60:40687)
0928 12:31:11.998145 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:11.998146 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:11.998275 (+ 129us) server_negotiation.cc:407] Connection header received
0928 12:31:11.999773 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:11.999775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:11.999797 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:11.999873 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:12.001454 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.002568 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.004333 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.004604 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.004682 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.009033 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:12.009034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:12.009035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:12.009042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:12.011764 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.012096 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.012097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.012097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.012151 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.012358 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.012359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.012360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.012491 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:12.012570 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.013250 (+ 680us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.013330 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:31:12.013491 15591 heartbeater.cc:511] Master 127.8.13.61:34599 requested a full tablet report, sending...
I20260928 12:31:12.013527 15590 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45429
I20260928 12:31:12.013569 15590 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:12.013628 15590 heartbeater.cc:511] Master 127.8.13.62:45429 requested a full tablet report, sending...
I20260928 12:31:12.013701 15593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:11.997966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:40687 (local address 127.8.13.3:58585)
0928 12:31:11.998203 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:11.998211 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:11.999575 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:12.000033 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:12.000041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:12.000924 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:12.001301 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.001341 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.002713 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.002714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.003857 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.003859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.004742 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.004743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.004903 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.007025 (+ 2122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:12.007029 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:12.008914 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:12.011830 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:12.011831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:12.011831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:12.012015 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:12.012223 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:12.012223 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:12.012224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:12.012280 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:12.013089 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:12.013089 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:12.013132 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 12:31:12.013483 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 12:31:12.013581 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":77,"thread_start_us":82,"threads_started":1}
I20260928 12:31:12.013971 14943 ts_manager.cc:194] Registered new tserver with Master: 8b411c034e084e3a9e811cb2aceea760 (127.8.13.3:46869)
I20260928 12:31:12.014003 14346 ts_manager.cc:194] Re-registered known tserver with Master: 8b411c034e084e3a9e811cb2aceea760 (127.8.13.3:46869)
I20260928 12:31:12.014456 14346 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:44185
I20260928 12:31:12.014580 15589 heartbeater.cc:347] Connected to a master server at 127.8.13.60:40687
I20260928 12:31:12.014617 15589 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:12.014678 15589 heartbeater.cc:511] Master 127.8.13.60:40687 requested a full tablet report, sending...
I20260928 12:31:12.015671 15137 ts_manager.cc:194] Registered new tserver with Master: 8b411c034e084e3a9e811cb2aceea760 (127.8.13.3:46869)
I20260928 12:31:12.016806 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.004896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.8.13.60:40687)
0928 12:31:12.004997 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.004999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.005182 (+ 183us) server_negotiation.cc:407] Connection header received
0928 12:31:12.005229 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.005230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.005243 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.005307 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:12.005928 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.006672 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.007859 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.008086 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.008146 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.009382 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:12.009383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:12.009384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:12.009390 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:12.012167 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.015568 (+ 3401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.015570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.015570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.015664 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.015863 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.015863 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.015864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.015995 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:12.016097 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.016609 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.016709 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":53,"threads_started":1}
I20260928 12:31:12.018613 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.004599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.8.13.62:45429)
0928 12:31:12.004797 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.004799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.004971 (+ 172us) server_negotiation.cc:407] Connection header received
0928 12:31:12.004998 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.004999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.005012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.005088 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:12.005939 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.006676 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.008653 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.008996 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.009056 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.015564 (+ 6508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:12.015565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:12.015565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:12.015572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:12.017482 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.017865 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.017867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.017867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.017936 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.018116 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.018116 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.018116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.018208 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:12.018273 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.018440 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.018513 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":117,"threads_started":1}
I20260928 12:31:12.019481 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.004337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.8.13.61:34599)
0928 12:31:12.005063 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.005065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.005246 (+ 181us) server_negotiation.cc:407] Connection header received
0928 12:31:12.005268 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.005269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.005279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.005327 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:12.013462 (+ 8135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.014165 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.015140 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.015419 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.015461 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.016841 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:12.016843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:12.016843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:12.016851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:12.018459 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.018808 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.018809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.018809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.018862 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.019018 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.019019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.019019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.019126 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:12.019213 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.019294 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.019367 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":75,"threads_started":1}
I20260928 12:31:12.020278 14346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:31:12.021912 14346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:31:12.022681 14905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: NON_VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: true } } }
I20260928 12:31:12.022781 14905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:12.023525 15607 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f: attempting to promote NON_VOTER b7fbc38338d64aa198c42bfdbb4698f4 to VOTER
I20260928 12:31:12.023671 15607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } }
I20260928 12:31:12.024068 15147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 LEARNER]: Refusing update from remote peer 2726055b354a40239b82c2c722552b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 12:31:12.024075 14953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Refusing update from remote peer 2726055b354a40239b82c2c722552b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 12:31:12.024242 14905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 12:31:12.024338 14904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 12:31:12.025156 15607 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b7fbc38338d64aa198c42bfdbb4698f4 (127.8.13.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:12.025167 15147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b7fbc38338d64aa198c42bfdbb4698f4 (127.8.13.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:12.025260 14953 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b7fbc38338d64aa198c42bfdbb4698f4 (127.8.13.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:12.026211 15176 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:12.026309 15176 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:31:12.026270 15099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:12.026337 15099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:12.026417 15099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:12.026391 15101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:12.026463 15099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:12.026469 15101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:31:12.026530 15607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2726055b354a40239b82c2c722552b5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:12.026576 15607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:12.031839 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.028080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39495 (local address 127.0.0.1:40662)
0928 12:31:12.028129 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:12.028141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:12.028182 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:12.028537 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:12.028538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:12.028540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:12.028596 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.028598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.029580 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.029581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.030569 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.030572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.031605 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.031606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.031664 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.031668 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:12.031715 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:31:12.031761 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:31:12.031848 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.028188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.8.13.1:39495)
0928 12:31:12.028346 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.028352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.028373 (+ 21us) server_negotiation.cc:407] Connection header received
0928 12:31:12.028418 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.028420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.028437 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.028665 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:12.028720 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.029517 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.030891 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.031534 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.031595 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.031674 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.031727 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.031763 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260928 12:31:12.032637 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.028479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.8.13.3:46869)
0928 12:31:12.028634 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.028639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.028648 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:12.028687 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.028689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.028706 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.028926 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:12.028977 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.030047 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.031277 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.032167 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.032246 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.032428 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.032457 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.032504 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260928 12:31:12.032850 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.028120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:44661 (local address 127.0.0.1:48994)
0928 12:31:12.028260 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:12.028278 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:12.028322 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:12.029087 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:12.029088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:12.029090 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:12.029133 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.029134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.030472 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.030474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.031224 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.031226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.032336 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.032337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.032531 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.032543 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 12:31:12.032713 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 12:31:12.032770 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260928 12:31:12.032832 15250 tablet_service.cc:1467] Processing CreateTablet for tablet ee34f029185c4b1692140ab6aa7a1632 (DEFAULT_TABLE table=test-table [id=9fc586c8ad86418db562c9a89bde15a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:12.033179 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.028304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:46869 (local address 127.0.0.1:58180)
0928 12:31:12.028455 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:12.028463 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:12.028500 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:12.028804 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:12.028805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:12.028807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:12.028854 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.028856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.030127 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.030128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.031191 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.031193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.032249 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.032250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.032312 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.032316 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:12.032359 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:31:12.033119 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":118,"threads_started":1}
I20260928 12:31:12.033254 15250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee34f029185c4b1692140ab6aa7a1632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:12.033521 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.028592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.8.13.2:44661)
0928 12:31:12.028903 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.028907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.028916 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:12.028976 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.028978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.028996 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.029072 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:12.029202 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.030376 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.031329 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.032238 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.032444 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.032617 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.033266 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.033436 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":144,"threads_started":1}
I20260928 12:31:12.034134 15523 tablet_service.cc:1467] Processing CreateTablet for tablet ee34f029185c4b1692140ab6aa7a1632 (DEFAULT_TABLE table=test-table [id=9fc586c8ad86418db562c9a89bde15a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:12.034515 15523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee34f029185c4b1692140ab6aa7a1632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:12.034415 15387 tablet_service.cc:1467] Processing CreateTablet for tablet ee34f029185c4b1692140ab6aa7a1632 (DEFAULT_TABLE table=test-table [id=9fc586c8ad86418db562c9a89bde15a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:12.034755 15387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee34f029185c4b1692140ab6aa7a1632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:12.037166 15619 tablet_bootstrap.cc:492] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3: Bootstrap starting.
I20260928 12:31:12.038043 15620 tablet_bootstrap.cc:492] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: Bootstrap starting.
I20260928 12:31:12.038828 15620 tablet_bootstrap.cc:654] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:12.039016 15619 tablet_bootstrap.cc:654] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:12.039471 15621 tablet_bootstrap.cc:492] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760: Bootstrap starting.
I20260928 12:31:12.039760 15620 log.cc:824] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:12.040103 15619 log.cc:824] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:12.040159 15621 tablet_bootstrap.cc:654] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:12.040732 15620 tablet_bootstrap.cc:492] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: No bootstrap required, opened a new log
I20260928 12:31:12.040820 15620 ts_tablet_manager.cc:1397] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:31:12.041010 15621 log.cc:824] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:12.041139 15619 tablet_bootstrap.cc:492] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3: No bootstrap required, opened a new log
I20260928 12:31:12.041211 15619 ts_tablet_manager.cc:1397] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 12:31:12.041921 15621 tablet_bootstrap.cc:492] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760: No bootstrap required, opened a new log
I20260928 12:31:12.041992 15621 ts_tablet_manager.cc:1397] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:31:12.045694 15619 raft_consensus.cc:348] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:12.045787 15621 raft_consensus.cc:348] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:12.045951 15619 raft_consensus.cc:374] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:12.045989 15619 raft_consensus.cc:729] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8acc77419c94659863723a919a041f3, State: Initialized, Role: FOLLOWER
I20260928 12:31:12.045990 15621 raft_consensus.cc:374] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:12.045902 15620 raft_consensus.cc:348] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:12.046026 15621 raft_consensus.cc:729] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b411c034e084e3a9e811cb2aceea760, State: Initialized, Role: FOLLOWER
I20260928 12:31:12.046044 15620 raft_consensus.cc:374] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:12.046072 15620 raft_consensus.cc:729] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33e75f6c55a4f39ae62d0547553141f, State: Initialized, Role: FOLLOWER
I20260928 12:31:12.046150 15619 consensus_queue.cc:260] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:12.046162 15621 consensus_queue.cc:260] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:12.046181 15620 consensus_queue.cc:260] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:12.046537 15620 ts_tablet_manager.cc:1428] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 12:31:12.046537 15619 ts_tablet_manager.cc:1428] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260928 12:31:12.046540 15621 ts_tablet_manager.cc:1428] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 12:31:12.046667 15590 heartbeater.cc:503] Master 127.8.13.62:45429 was elected leader, sending a full tablet report...
I20260928 12:31:12.047423 15454 heartbeater.cc:503] Master 127.8.13.62:45429 was elected leader, sending a full tablet report...
I20260928 12:31:12.047603 15317 heartbeater.cc:503] Master 127.8.13.62:45429 was elected leader, sending a full tablet report...
W20260928 12:31:12.084420 15456 tablet.cc:2369] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:31:12.137863 15627 raft_consensus.cc:482] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:12.138001 15627 raft_consensus.cc:504] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:12.138582 15627 leader_election.cc:288] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c33e75f6c55a4f39ae62d0547553141f (127.8.13.2:44661), 8b411c034e084e3a9e811cb2aceea760 (127.8.13.3:46869)
I20260928 12:31:12.141757 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.138757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:44661 (local address 127.8.13.1:59415)
0928 12:31:12.138823 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:12.138836 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:12.138904 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:12.139071 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:12.139072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:12.139074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:12.139278 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.139281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.139923 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.139924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.140793 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.140795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.141407 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.141409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.141559 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.141563 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:12.141624 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 12:31:12.141673 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 12:31:12.141777 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.138858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:59415 (local address 127.8.13.2:44661)
0928 12:31:12.138936 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.138938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.138964 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:31:12.138994 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.138995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.139008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.139053 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:12.139363 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.139839 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.140948 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.141315 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.141388 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.141398 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.141675 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.141720 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 12:31:12.142266 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.138964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:59777 (local address 127.8.13.3:46869)
0928 12:31:12.139009 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.139010 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.139018 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:12.139045 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.139046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.139058 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.139112 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:12.139362 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.139836 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.140768 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.141737 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.141826 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.141837 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.142150 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.142198 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:31:12.142292 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.138858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:46869 (local address 127.8.13.1:59777)
0928 12:31:12.138909 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:12.138919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:12.138955 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:12.139122 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:12.139122 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:12.139123 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:12.139277 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.139279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.139917 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.139918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.140669 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:12.140672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.141858 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.141859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:12.142061 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.142065 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:12.142133 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:31:12.142184 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 12:31:12.142284 15407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee34f029185c4b1692140ab6aa7a1632" candidate_uuid: "b8acc77419c94659863723a919a041f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33e75f6c55a4f39ae62d0547553141f" is_pre_election: true
I20260928 12:31:12.142452 15407 raft_consensus.cc:2454] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8acc77419c94659863723a919a041f3 in term 0.
I20260928 12:31:12.142709 15205 leader_election.cc:302] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8acc77419c94659863723a919a041f3, c33e75f6c55a4f39ae62d0547553141f; no voters:
I20260928 12:31:12.142752 15543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee34f029185c4b1692140ab6aa7a1632" candidate_uuid: "b8acc77419c94659863723a919a041f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b411c034e084e3a9e811cb2aceea760" is_pre_election: true
I20260928 12:31:12.142853 15627 raft_consensus.cc:2781] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:31:12.142890 15543 raft_consensus.cc:2454] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8acc77419c94659863723a919a041f3 in term 0.
I20260928 12:31:12.142915 15627 raft_consensus.cc:482] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:31:12.142937 15627 raft_consensus.cc:3037] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:12.144225 15627 raft_consensus.cc:504] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:12.144345 15627 leader_election.cc:288] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [CANDIDATE]: Term 1 election: Requested vote from peers c33e75f6c55a4f39ae62d0547553141f (127.8.13.2:44661), 8b411c034e084e3a9e811cb2aceea760 (127.8.13.3:46869)
I20260928 12:31:12.144591 15543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee34f029185c4b1692140ab6aa7a1632" candidate_uuid: "b8acc77419c94659863723a919a041f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b411c034e084e3a9e811cb2aceea760"
I20260928 12:31:12.144665 15543 raft_consensus.cc:3037] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:12.144695 15407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee34f029185c4b1692140ab6aa7a1632" candidate_uuid: "b8acc77419c94659863723a919a041f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33e75f6c55a4f39ae62d0547553141f"
I20260928 12:31:12.144793 15407 raft_consensus.cc:3037] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:12.145550 15407 raft_consensus.cc:2454] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8acc77419c94659863723a919a041f3 in term 1.
I20260928 12:31:12.145740 15205 leader_election.cc:302] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8acc77419c94659863723a919a041f3, c33e75f6c55a4f39ae62d0547553141f; no voters:
I20260928 12:31:12.145857 15627 raft_consensus.cc:2781] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:12.145943 15543 raft_consensus.cc:2454] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8acc77419c94659863723a919a041f3 in term 1.
I20260928 12:31:12.146097 15627 raft_consensus.cc:686] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 1 LEADER]: Becoming Leader. State: Replica: b8acc77419c94659863723a919a041f3, State: Running, Role: LEADER
I20260928 12:31:12.146206 15627 consensus_queue.cc:237] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:12.147207 14345 catalog_manager.cc:5495] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8acc77419c94659863723a919a041f3 (127.8.13.1). New cstate: current_term: 1 leader_uuid: "b8acc77419c94659863723a919a041f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } health_report { overall_health: UNKNOWN } } }
I20260928 12:31:12.187196 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.183339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.8.13.1:39495)
0928 12:31:12.183412 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.183414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.183430 (+ 16us) server_negotiation.cc:407] Connection header received
0928 12:31:12.183516 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.183518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.183696 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.183831 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:31:12.184216 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.184774 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.185942 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.186384 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.186469 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.186763 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:31:12.186944 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:31:12.187043 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.187067 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.187101 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:31:12.189445 15543 raft_consensus.cc:1261] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760 [term 1 FOLLOWER]: Refusing update from remote peer b8acc77419c94659863723a919a041f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:12.189983 15407 raft_consensus.cc:1261] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f [term 1 FOLLOWER]: Refusing update from remote peer b8acc77419c94659863723a919a041f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:12.189956 15628 consensus_queue.cc:1035] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:12.190471 15628 consensus_queue.cc:1035] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:12.194247 15631 mvcc.cc:205] Tried to move back new op lower bound from 7334292161284550656 to 7334292161111363584. Current Snapshot: MvccSnapshot[applied={T|T < 7334292161284550656}]
I20260928 12:31:12.195308 15632 mvcc.cc:205] Tried to move back new op lower bound from 7334292161284550656 to 7334292161111363584. Current Snapshot: MvccSnapshot[applied={T|T < 7334292161284550656}]
I20260928 12:31:12.195981 15634 mvcc.cc:205] Tried to move back new op lower bound from 7334292161284550656 to 7334292161111363584. Current Snapshot: MvccSnapshot[applied={T|T < 7334292161284550656}]
I20260928 12:31:12.198137 14345 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:12.198247 14345 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:12.199631 14345 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2726055b354a40239b82c2c722552b5f: Sending DeleteTablet for 3 replicas of tablet ee34f029185c4b1692140ab6aa7a1632
I20260928 12:31:12.200086 15250 tablet_service.cc:1514] Processing DeleteTablet for tablet ee34f029185c4b1692140ab6aa7a1632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:12 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:40662
I20260928 12:31:12.200160 15523 tablet_service.cc:1514] Processing DeleteTablet for tablet ee34f029185c4b1692140ab6aa7a1632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:12 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:58180
I20260928 12:31:12.200467 15640 tablet_replica.cc:330] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3: stopping tablet replica
I20260928 12:31:12.200510 15387 tablet_service.cc:1514] Processing DeleteTablet for tablet ee34f029185c4b1692140ab6aa7a1632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:12 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:48994
I20260928 12:31:12.200580 15640 raft_consensus.cc:2229] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:31:12.200680 15641 tablet_replica.cc:330] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760: stopping tablet replica
I20260928 12:31:12.200690 15640 raft_consensus.cc:2258] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:12.200838 15641 raft_consensus.cc:2229] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:12.201057 15641 raft_consensus.cc:2258] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:12.201203 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 14314
I20260928 12:31:12.201226 15640 ts_tablet_manager.cc:1905] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:12.202324 15641 ts_tablet_manager.cc:1905] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:12.203807 15640 ts_tablet_manager.cc:1918] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:12.203881 15640 log.cc:1196] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-0/wal/wals/ee34f029185c4b1692140ab6aa7a1632
I20260928 12:31:12.204130 15640 ts_tablet_manager.cc:1939] T ee34f029185c4b1692140ab6aa7a1632 P b8acc77419c94659863723a919a041f3: Deleting consensus metadata
I20260928 12:31:12.204138 15641 ts_tablet_manager.cc:1918] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:12.204316 15641 log.cc:1196] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/ee34f029185c4b1692140ab6aa7a1632
I20260928 12:31:12.204617 15641 ts_tablet_manager.cc:1939] T ee34f029185c4b1692140ab6aa7a1632 P 8b411c034e084e3a9e811cb2aceea760: Deleting consensus metadata
I20260928 12:31:12.206199 15643 tablet_replica.cc:330] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: stopping tablet replica
I20260928 12:31:12.206341 15643 raft_consensus.cc:2229] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:12.206475 15643 raft_consensus.cc:2258] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:12.206928 15643 ts_tablet_manager.cc:1905] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 12:31:12.206969 15477 connection.cc:535] server connection from 127.0.0.1:58180 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 12:31:12.207464 15205 connection.cc:535] server connection from 127.0.0.1:40662 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:31:12.208756 15643 ts_tablet_manager.cc:1918] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:12.208851 15643 log.cc:1196] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/ee34f029185c4b1692140ab6aa7a1632
I20260928 12:31:12.209131 15643 ts_tablet_manager.cc:1939] T ee34f029185c4b1692140ab6aa7a1632 P c33e75f6c55a4f39ae62d0547553141f: Deleting consensus metadata
W20260928 12:31:12.209528 15341 connection.cc:441] server connection from 127.0.0.1:48994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48994 (request call id 1) could send its response
I20260928 12:31:12.216032 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.201056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.8.13.60:40687)
0928 12:31:12.201106 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.201109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.206018 (+ 4909us) server_negotiation.cc:407] Connection header received
0928 12:31:12.206115 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.206116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.206133 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.206188 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:12.207824 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.208595 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.209959 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.210275 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.210339 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.211740 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:12.211742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:12.211742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:12.211749 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:12.213917 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.214787 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.214788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.214788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.214846 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.215043 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.215044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.215044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.215177 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:12.215247 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.215837 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.215929 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:31:12.218276 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.209525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.8.13.61:34599)
0928 12:31:12.209638 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.209640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.210273 (+ 633us) server_negotiation.cc:407] Connection header received
0928 12:31:12.210365 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.210366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.210381 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.210408 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:12.211512 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.212455 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.213787 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.214134 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.214176 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.215412 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:12.215414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:12.215414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:12.215420 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:12.216930 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.217316 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.217318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.217318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.217376 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.217597 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.217598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.217599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.217752 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:12.217872 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.218093 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.218183 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 12:31:12.218636 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:12.209188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.8.13.60:40687)
0928 12:31:12.209233 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:12.209235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:12.210055 (+ 820us) server_negotiation.cc:407] Connection header received
0928 12:31:12.210143 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:12.210145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:12.210157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:12.210216 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:12.211496 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.212242 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.213293 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:12.213586 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:12.213624 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:12.215232 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:12.215234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:12.215234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:12.215240 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:12.217460 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.217726 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.217727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.217727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.217776 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:12.218016 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:12.218017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:12.218017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:12.218130 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:12.218213 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:12.218475 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 12:31:12.218550 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:31:13.195886 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.195570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.2:42393)
0928 12:31:13.195740 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.195826 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
W20260928 12:31:13.196281 15454 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:45429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
I20260928 12:31:13.196878 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.196623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.1:53895)
0928 12:31:13.196763 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.196833 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":56,"threads_started":1}
W20260928 12:31:13.197292 15317 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:45429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
I20260928 12:31:13.198865 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.198611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.3:50817)
0928 12:31:13.198714 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.198803 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":53,"threads_started":1}
W20260928 12:31:13.199128 15590 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:45429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
I20260928 12:31:13.700394 15654 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2726055b354a40239b82c2c722552b5f)
I20260928 12:31:13.700513 15655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2726055b354a40239b82c2c722552b5f)
I20260928 12:31:13.700603 15654 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } }
I20260928 12:31:13.700649 15655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } }
I20260928 12:31:13.700836 15655 leader_election.cc:288] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429), b7fbc38338d64aa198c42bfdbb4698f4 (127.8.13.60:40687)
I20260928 12:31:13.700929 15654 leader_election.cc:288] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429), 73a8920c213d4947a6504ea84885e8fb (127.8.13.61:34599)
I20260928 12:31:13.701221 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.701020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49376)
0928 12:31:13.701136 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.701177 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260928 12:31:13.701251 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.700997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49366)
0928 12:31:13.701158 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.701213 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":55,"threads_started":1}
W20260928 12:31:13.701318 15123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
W20260928 12:31:13.701458 14929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
I20260928 12:31:13.701974 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.701875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49392)
0928 12:31:13.701910 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.701944 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:31:13.702056 14929 leader_election.cc:334] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429): Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
I20260928 12:31:13.702207 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.702064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49406)
0928 12:31:13.702156 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.702180 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 12:31:13.702271 15123 leader_election.cc:334] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429): Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
I20260928 12:31:13.704216 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.701342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.8.13.60:40687)
0928 12:31:13.701452 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:13.701453 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:13.701461 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:13.701485 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:13.701486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:13.701496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:13.701816 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:13.701850 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:13.702500 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:13.703502 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:13.703889 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:13.703948 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:13.703960 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:13.704074 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 12:31:13.704118 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":43,"threads_started":1}
I20260928 12:31:13.704216 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.701142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:40687 (local address 127.0.0.1:46564)
0928 12:31:13.701305 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.701314 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:13.701366 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:13.701588 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:13.701590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:13.701593 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:13.701677 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:13.701679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:13.702579 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:13.702580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:13.703420 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:13.703422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:13.703958 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:13.703959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:13.704014 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:13.704018 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:13.704064 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 12:31:13.704108 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":52,"threads_started":1}
I20260928 12:31:13.704226 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.701050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:34599 (local address 127.0.0.1:57484)
0928 12:31:13.701188 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.701199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:13.701278 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:13.701605 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:13.701606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:13.701610 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:13.701684 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:13.701686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:13.702620 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:13.702621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:13.703447 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:13.703450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:13.703971 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:13.703972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:13.704032 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:13.704035 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:13.704081 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 12:31:13.704128 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":41,"threads_started":1}
I20260928 12:31:13.704222 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.701108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.8.13.61:34599)
0928 12:31:13.701459 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:13.701460 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:13.701468 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:13.701490 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:13.701491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:13.701500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:13.701766 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:13.701821 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:13.702540 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:13.703535 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:13.703901 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:13.703959 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:13.703971 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:13.704096 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 12:31:13.704134 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":42,"threads_started":1}
I20260928 12:31:13.704655 14953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "73a8920c213d4947a6504ea84885e8fb" is_pre_election: true
I20260928 12:31:13.704654 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73a8920c213d4947a6504ea84885e8fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" is_pre_election: true
I20260928 12:31:13.704782 15147 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a8920c213d4947a6504ea84885e8fb in term 1.
I20260928 12:31:13.704779 14953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7fbc38338d64aa198c42bfdbb4698f4 in term 1.
I20260928 12:31:13.704974 14929 leader_election.cc:302] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73a8920c213d4947a6504ea84885e8fb, b7fbc38338d64aa198c42bfdbb4698f4; no voters: 2726055b354a40239b82c2c722552b5f
I20260928 12:31:13.704972 15122 leader_election.cc:302] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73a8920c213d4947a6504ea84885e8fb, b7fbc38338d64aa198c42bfdbb4698f4; no voters: 2726055b354a40239b82c2c722552b5f
I20260928 12:31:13.705061 15655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 12:31:13.705080 15654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 12:31:13.705101 15655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2726055b354a40239b82c2c722552b5f)
I20260928 12:31:13.705116 15654 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2726055b354a40239b82c2c722552b5f)
I20260928 12:31:13.705128 15655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:31:13.705137 15654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:31:13.705842 15655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } }
I20260928 12:31:13.705834 15654 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } }
I20260928 12:31:13.705976 15655 leader_election.cc:288] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 2 election: Requested vote from peers 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429), b7fbc38338d64aa198c42bfdbb4698f4 (127.8.13.60:40687)
I20260928 12:31:13.705987 15654 leader_election.cc:288] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [CANDIDATE]: Term 2 election: Requested vote from peers 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429), 73a8920c213d4947a6504ea84885e8fb (127.8.13.61:34599)
I20260928 12:31:13.706148 14953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "73a8920c213d4947a6504ea84885e8fb"
I20260928 12:31:13.706197 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.706077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49412)
0928 12:31:13.706141 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.706169 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 12:31:13.706207 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.706079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49410)
0928 12:31:13.706157 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.706183 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 12:31:13.706233 14953 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b7fbc38338d64aa198c42bfdbb4698f4 in current term 2: Already voted for candidate 73a8920c213d4947a6504ea84885e8fb in this term.
I20260928 12:31:13.706300 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73a8920c213d4947a6504ea84885e8fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b7fbc38338d64aa198c42bfdbb4698f4"
I20260928 12:31:13.706385 15147 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 73a8920c213d4947a6504ea84885e8fb in current term 2: Already voted for candidate b7fbc38338d64aa198c42bfdbb4698f4 in this term.
I20260928 12:31:13.706542 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.706451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49426)
0928 12:31:13.706488 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.706517 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 12:31:13.706590 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:13.706499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49442)
0928 12:31:13.706535 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:13.706563 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 12:31:13.706609 15123 leader_election.cc:334] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429): Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
I20260928 12:31:13.706638 15123 leader_election.cc:302] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7fbc38338d64aa198c42bfdbb4698f4; no voters: 2726055b354a40239b82c2c722552b5f, 73a8920c213d4947a6504ea84885e8fb
W20260928 12:31:13.706660 14929 leader_election.cc:334] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429): Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
I20260928 12:31:13.706694 14929 leader_election.cc:302] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73a8920c213d4947a6504ea84885e8fb; no voters: 2726055b354a40239b82c2c722552b5f, b7fbc38338d64aa198c42bfdbb4698f4
I20260928 12:31:13.706698 15654 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 12:31:13.706748 15655 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 12:31:14.196950 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:14.196708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.2:46133)
0928 12:31:14.196842 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:14.196900 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260928 12:31:14.197738 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:14.197574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.1:46871)
0928 12:31:14.197673 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:14.197704 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":48,"threads_started":1}
I20260928 12:31:14.199642 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:14.199436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.3:49815)
0928 12:31:14.199555 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:14.199596 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260928 12:31:15.197687 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.197454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.2:47135)
0928 12:31:15.197591 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.197639 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20260928 12:31:15.198263 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.198103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.1:39159)
0928 12:31:15.198201 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.198229 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":45,"threads_started":1}
I20260928 12:31:15.200189 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.200019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.3:45583)
0928 12:31:15.200112 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.200151 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":52,"threads_started":1}
I20260928 12:31:15.359439 15670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:15.359551 15670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } }
I20260928 12:31:15.359717 15670 leader_election.cc:288] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429), b7fbc38338d64aa198c42bfdbb4698f4 (127.8.13.60:40687)
I20260928 12:31:15.360054 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.359860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49676)
0928 12:31:15.359974 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.360018 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260928 12:31:15.360169 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73a8920c213d4947a6504ea84885e8fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" is_pre_election: true
I20260928 12:31:15.360267 15147 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a8920c213d4947a6504ea84885e8fb in term 2.
I20260928 12:31:15.360420 14929 leader_election.cc:302] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a8920c213d4947a6504ea84885e8fb, b7fbc38338d64aa198c42bfdbb4698f4; no voters:
I20260928 12:31:15.360478 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.360373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49688)
0928 12:31:15.360421 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.360451 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:31:15.360504 15670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 12:31:15.360545 15670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260928 12:31:15.360550 14929 leader_election.cc:334] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429): Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
I20260928 12:31:15.360589 15670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 2 FOLLOWER]: Advancing to term 3
I20260928 12:31:15.361290 15670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } }
I20260928 12:31:15.361392 15670 leader_election.cc:288] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 3 election: Requested vote from peers 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429), b7fbc38338d64aa198c42bfdbb4698f4 (127.8.13.60:40687)
I20260928 12:31:15.361614 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.361514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49700)
0928 12:31:15.361558 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.361587 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 12:31:15.361668 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73a8920c213d4947a6504ea84885e8fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b7fbc38338d64aa198c42bfdbb4698f4"
I20260928 12:31:15.361747 15147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 2 FOLLOWER]: Advancing to term 3
I20260928 12:31:15.361924 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.361850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49712)
0928 12:31:15.361878 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.361903 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 12:31:15.361994 14929 leader_election.cc:334] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429): Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
I20260928 12:31:15.362426 15147 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a8920c213d4947a6504ea84885e8fb in term 3.
I20260928 12:31:15.362602 14929 leader_election.cc:302] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73a8920c213d4947a6504ea84885e8fb, b7fbc38338d64aa198c42bfdbb4698f4; no voters: 2726055b354a40239b82c2c722552b5f
I20260928 12:31:15.362713 15670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 3 FOLLOWER]: Leader election won for term 3
I20260928 12:31:15.362843 15670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [term 3 LEADER]: Becoming Leader. State: Replica: 73a8920c213d4947a6504ea84885e8fb, State: Running, Role: LEADER
I20260928 12:31:15.362929 15670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } }
I20260928 12:31:15.363274 15673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73a8920c213d4947a6504ea84885e8fb. Latest consensus state: current_term: 3 leader_uuid: "73a8920c213d4947a6504ea84885e8fb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:15.363368 15673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:15.363592 15675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:15.364006 15675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:15.364073 15675 catalog_manager.cc:1259] Loaded cluster ID: ded15d47e7974d518d3d52c27a1cda2a
I20260928 12:31:15.364091 15675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:31:15.364277 15675 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:15.364343 15675 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: Loaded TSK: 0
I20260928 12:31:15.364480 15675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:31:15.394109 14943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:31:15.395713 14943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:31:15.396749 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.396603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49736)
0928 12:31:15.396674 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.396709 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 12:31:15.396867 15147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [term 3 FOLLOWER]: Refusing update from remote peer 73a8920c213d4947a6504ea84885e8fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 12:31:15.397047 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.396970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.0.0.1:49748)
0928 12:31:15.396998 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.397031 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 12:31:15.397069 15670 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 12:31:15.397202 14929 consensus_peers.cc:487] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb -> Peer 2726055b354a40239b82c2c722552b5f (127.8.13.62:45429): Couldn't send request to peer 2726055b354a40239b82c2c722552b5f. Status: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 12:31:15.397676 15677 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73a8920c213d4947a6504ea84885e8fb. Latest consensus state: current_term: 3 leader_uuid: "73a8920c213d4947a6504ea84885e8fb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:15.397820 15677 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:31:15.398182 15670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "73a8920c213d4947a6504ea84885e8fb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:15.398272 15670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:15.398312 15678 mvcc.cc:205] Tried to move back new op lower bound from 7334292174422962176 to 7334292174287056896. Current Snapshot: MvccSnapshot[applied={T|T < 7334292174422962176}]
I20260928 12:31:15.398451 15677 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "73a8920c213d4947a6504ea84885e8fb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2726055b354a40239b82c2c722552b5f" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45429 } } peers { permanent_uuid: "73a8920c213d4947a6504ea84885e8fb" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 34599 } attrs { promote: false } } peers { permanent_uuid: "b7fbc38338d64aa198c42bfdbb4698f4" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 40687 } attrs { promote: false } } }
I20260928 12:31:15.398610 15677 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fbc38338d64aa198c42bfdbb4698f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:31:15.403455 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.400096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.8.13.1:39495)
0928 12:31:15.400418 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:15.400420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:15.400428 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:15.400453 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:15.400454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:15.400464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:15.400632 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:15.400667 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.401475 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.402772 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.403147 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.403207 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:15.403258 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:15.403345 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 12:31:15.403384 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":67,"threads_started":1}
I20260928 12:31:15.403506 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.400111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:46869 (local address 127.0.0.1:58182)
0928 12:31:15.400295 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.400304 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:15.400348 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:15.400592 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:15.400593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:15.400595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:15.400638 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:15.400640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.401344 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.401345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:15.402555 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:15.402558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.403294 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.403294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:15.403352 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:15.403355 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:15.403392 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 12:31:15.403427 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":20,"thread_start_us":67,"threads_started":1}
I20260928 12:31:15.403517 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.399958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:44661 (local address 127.0.0.1:48998)
0928 12:31:15.400086 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.400095 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:15.400140 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:15.400513 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:15.400514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:15.400517 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:15.400575 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:15.400576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.401571 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.401572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:15.402485 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:15.402487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.403220 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.403221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:15.403291 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:15.403295 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:15.403323 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 12:31:15.403369 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260928 12:31:15.403563 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.400273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.8.13.2:44661)
0928 12:31:15.400369 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:15.400371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:15.400378 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:31:15.400405 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:15.400406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:15.400414 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:15.400633 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:15.400667 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.401475 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.402549 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.403118 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.403431 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:15.403444 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:15.403461 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 12:31:15.403500 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":49,"threads_started":1}
I20260928 12:31:15.403627 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.399912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39495 (local address 127.0.0.1:40692)
0928 12:31:15.399952 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.399961 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:15.400029 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:15.400547 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:15.400548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:15.400549 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:15.400574 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:15.400576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.401548 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.401549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:15.402703 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:15.402705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.403216 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.403217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:15.403283 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:15.403287 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:15.403339 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:31:15.403384 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":3}
I20260928 12:31:15.403723 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.400181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.8.13.3:46869)
0928 12:31:15.400439 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:15.400441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:15.400448 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:31:15.400472 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:15.400473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:15.400482 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:15.400544 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:15.400701 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.401283 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.402627 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.403185 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.403254 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:15.403267 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:15.403606 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 12:31:15.403656 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":61,"threads_started":1}
I20260928 12:31:15.404217 15523 tablet_service.cc:1467] Processing CreateTablet for tablet e514702b11404a37b61bdddb52cd089d (DEFAULT_TABLE table=test-table [id=3ebc2f726b1f4321aeecac0e5b89a527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:15.404258 15387 tablet_service.cc:1467] Processing CreateTablet for tablet e514702b11404a37b61bdddb52cd089d (DEFAULT_TABLE table=test-table [id=3ebc2f726b1f4321aeecac0e5b89a527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:15.404299 15250 tablet_service.cc:1467] Processing CreateTablet for tablet e514702b11404a37b61bdddb52cd089d (DEFAULT_TABLE table=test-table [id=3ebc2f726b1f4321aeecac0e5b89a527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:15.404364 15523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e514702b11404a37b61bdddb52cd089d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:15.404389 15250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e514702b11404a37b61bdddb52cd089d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:15.404388 15387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e514702b11404a37b61bdddb52cd089d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:15.405722 15689 tablet_bootstrap.cc:492] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3: Bootstrap starting.
I20260928 12:31:15.405915 15687 tablet_bootstrap.cc:492] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f: Bootstrap starting.
I20260928 12:31:15.406126 15689 tablet_bootstrap.cc:654] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:15.406231 15687 tablet_bootstrap.cc:654] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:15.406379 15688 tablet_bootstrap.cc:492] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760: Bootstrap starting.
I20260928 12:31:15.406608 15689 tablet_bootstrap.cc:492] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3: No bootstrap required, opened a new log
I20260928 12:31:15.406651 15689 ts_tablet_manager.cc:1397] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:15.406698 15688 tablet_bootstrap.cc:654] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:15.406735 15689 raft_consensus.cc:348] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:15.406791 15689 raft_consensus.cc:374] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:15.406804 15689 raft_consensus.cc:729] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8acc77419c94659863723a919a041f3, State: Initialized, Role: FOLLOWER
I20260928 12:31:15.406839 15689 consensus_queue.cc:260] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:15.406997 15689 ts_tablet_manager.cc:1428] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:15.407241 15688 tablet_bootstrap.cc:492] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760: No bootstrap required, opened a new log
I20260928 12:31:15.407330 15688 ts_tablet_manager.cc:1397] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:15.407473 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.407238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.1:42549)
0928 12:31:15.407409 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.407445 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260928 12:31:15.407444 15688 raft_consensus.cc:348] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:15.407502 15688 raft_consensus.cc:374] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:15.407521 15688 raft_consensus.cc:729] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b411c034e084e3a9e811cb2aceea760, State: Initialized, Role: FOLLOWER
I20260928 12:31:15.407580 15318 heartbeater.cc:503] Master 127.8.13.61:34599 was elected leader, sending a full tablet report...
I20260928 12:31:15.407574 15688 consensus_queue.cc:260] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:15.407732 15688 ts_tablet_manager.cc:1428] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:15.407886 15687 tablet_bootstrap.cc:492] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f: No bootstrap required, opened a new log
I20260928 12:31:15.407930 15687 ts_tablet_manager.cc:1397] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:31:15.408135 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.408035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.1:43421)
0928 12:31:15.408080 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.408109 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:31:15.408061 15687 raft_consensus.cc:348] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:15.408185 15687 raft_consensus.cc:374] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:15.408209 15687 raft_consensus.cc:729] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33e75f6c55a4f39ae62d0547553141f, State: Initialized, Role: FOLLOWER
I20260928 12:31:15.408339 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.408228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.3:44181)
0928 12:31:15.408291 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.408315 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 12:31:15.408382 15687 consensus_queue.cc:260] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:15.408558 15687 ts_tablet_manager.cc:1428] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:31:15.408844 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.408730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.2:33421)
0928 12:31:15.408771 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.408817 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 12:31:15.409052 15591 heartbeater.cc:503] Master 127.8.13.61:34599 was elected leader, sending a full tablet report...
I20260928 12:31:15.409209 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.409125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.3:60563)
0928 12:31:15.409159 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.409188 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 12:31:15.409281 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.409190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.2:46695)
0928 12:31:15.409224 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.409249 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 12:31:15.409313 15455 heartbeater.cc:503] Master 127.8.13.61:34599 was elected leader, sending a full tablet report...
W20260928 12:31:15.447883 15319 tablet.cc:2369] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:31:15.500440 15592 tablet.cc:2369] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:31:15.546175 15695 raft_consensus.cc:482] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:15.546276 15695 raft_consensus.cc:504] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:15.546608 15695 leader_election.cc:288] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8acc77419c94659863723a919a041f3 (127.8.13.1:39495), 8b411c034e084e3a9e811cb2aceea760 (127.8.13.3:46869)
I20260928 12:31:15.549603 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.546839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:37333 (local address 127.8.13.1:39495)
0928 12:31:15.546884 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:15.546885 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:15.546897 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:31:15.546952 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:15.546953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:15.546965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:15.547015 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:15.547207 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.547726 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.548901 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.549263 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.549320 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:15.549332 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:15.549493 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 12:31:15.549533 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:31:15.549602 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.546843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:57755 (local address 127.8.13.3:46869)
0928 12:31:15.546901 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:15.546903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:15.547014 (+ 111us) server_negotiation.cc:407] Connection header received
0928 12:31:15.547072 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:15.547073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:15.547083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:15.547134 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:15.547352 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.547859 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.548897 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.549271 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.549343 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:15.549354 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:15.549492 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 12:31:15.549529 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:31:15.549612 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.546768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39495 (local address 127.8.13.2:37333)
0928 12:31:15.546829 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.546840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:15.546890 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:15.547025 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:15.547025 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:15.547028 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:15.547116 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:15.547118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.547816 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.547817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:15.548784 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:15.548787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.549342 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.549343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:15.549417 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:15.549420 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:15.549471 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:31:15.549521 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":2}
I20260928 12:31:15.549621 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.546780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:46869 (local address 127.8.13.2:57755)
0928 12:31:15.546946 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.546955 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:15.547011 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:15.547173 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:15.547174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:15.547177 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:15.547240 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:15.547242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.547964 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.547965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:15.548786 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:15.548789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.549368 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.549369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:15.549427 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:15.549430 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:15.549477 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:31:15.549519 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":28,"thread_start_us":103,"threads_started":1}
I20260928 12:31:15.549947 15270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e514702b11404a37b61bdddb52cd089d" candidate_uuid: "c33e75f6c55a4f39ae62d0547553141f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8acc77419c94659863723a919a041f3" is_pre_election: true
I20260928 12:31:15.550086 15270 raft_consensus.cc:2454] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33e75f6c55a4f39ae62d0547553141f in term 0.
I20260928 12:31:15.550062 15543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e514702b11404a37b61bdddb52cd089d" candidate_uuid: "c33e75f6c55a4f39ae62d0547553141f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b411c034e084e3a9e811cb2aceea760" is_pre_election: true
I20260928 12:31:15.550132 15543 raft_consensus.cc:2454] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33e75f6c55a4f39ae62d0547553141f in term 0.
I20260928 12:31:15.550272 15343 leader_election.cc:302] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8acc77419c94659863723a919a041f3, c33e75f6c55a4f39ae62d0547553141f; no voters:
I20260928 12:31:15.550452 15695 raft_consensus.cc:2781] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:31:15.550495 15695 raft_consensus.cc:482] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:31:15.550509 15695 raft_consensus.cc:3037] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:15.551246 15695 raft_consensus.cc:504] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:15.551368 15695 leader_election.cc:288] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [CANDIDATE]: Term 1 election: Requested vote from peers b8acc77419c94659863723a919a041f3 (127.8.13.1:39495), 8b411c034e084e3a9e811cb2aceea760 (127.8.13.3:46869)
I20260928 12:31:15.551530 15543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e514702b11404a37b61bdddb52cd089d" candidate_uuid: "c33e75f6c55a4f39ae62d0547553141f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b411c034e084e3a9e811cb2aceea760"
I20260928 12:31:15.551602 15543 raft_consensus.cc:3037] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:15.551573 15270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e514702b11404a37b61bdddb52cd089d" candidate_uuid: "c33e75f6c55a4f39ae62d0547553141f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8acc77419c94659863723a919a041f3"
I20260928 12:31:15.551656 15270 raft_consensus.cc:3037] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:15.552162 15543 raft_consensus.cc:2454] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33e75f6c55a4f39ae62d0547553141f in term 1.
I20260928 12:31:15.552332 15270 raft_consensus.cc:2454] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33e75f6c55a4f39ae62d0547553141f in term 1.
I20260928 12:31:15.552346 15342 leader_election.cc:302] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b411c034e084e3a9e811cb2aceea760, c33e75f6c55a4f39ae62d0547553141f; no voters:
I20260928 12:31:15.552441 15695 raft_consensus.cc:2781] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:15.552590 15695 raft_consensus.cc:686] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [term 1 LEADER]: Becoming Leader. State: Replica: c33e75f6c55a4f39ae62d0547553141f, State: Running, Role: LEADER
I20260928 12:31:15.552654 15695 consensus_queue.cc:237] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } }
I20260928 12:31:15.552922 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.552817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.2:56533)
0928 12:31:15.552856 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.552892 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 12:31:15.553507 14943 catalog_manager.cc:5495] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f reported cstate change: term changed from 0 to 1, leader changed from <none> to c33e75f6c55a4f39ae62d0547553141f (127.8.13.2). New cstate: current_term: 1 leader_uuid: "c33e75f6c55a4f39ae62d0547553141f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33e75f6c55a4f39ae62d0547553141f" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 44661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 } health_report { overall_health: UNKNOWN } } }
I20260928 12:31:15.585841 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.583549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.8.13.2:44661)
0928 12:31:15.583602 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:15.583604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:15.583611 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:31:15.583647 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:15.583648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:15.583668 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:15.583733 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:31:15.583877 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.584399 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.585111 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:15.585294 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:15.585348 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:15.585493 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:31:15.585596 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:31:15.585656 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:15.585698 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 12:31:15.585720 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:31:15.586947 15543 raft_consensus.cc:1261] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760 [term 1 FOLLOWER]: Refusing update from remote peer c33e75f6c55a4f39ae62d0547553141f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:15.587042 15270 raft_consensus.cc:1261] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3 [term 1 FOLLOWER]: Refusing update from remote peer c33e75f6c55a4f39ae62d0547553141f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:15.587218 15695 consensus_queue.cc:1035] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b411c034e084e3a9e811cb2aceea760" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:15.587343 15697 consensus_queue.cc:1035] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8acc77419c94659863723a919a041f3" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:15.588294 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.588169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.1:59081)
0928 12:31:15.588224 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.588265 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 12:31:15.588359 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.588054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.3:55367)
0928 12:31:15.588261 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.588316 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260928 12:31:15.589846 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.589662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.2:45987)
0928 12:31:15.589748 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.589789 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 12:31:15.590875 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.590360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.2:42037)
0928 12:31:15.590812 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.590847 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435}
I20260928 12:31:15.590891 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.590786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.1:46547)
0928 12:31:15.590828 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.590864 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 12:31:15.591113 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.591000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45429 (local address 127.8.13.3:36183)
0928 12:31:15.591047 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.591079 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 12:31:15.592528 14943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:15.592625 14943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:15.593755 14943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73a8920c213d4947a6504ea84885e8fb: Sending DeleteTablet for 3 replicas of tablet e514702b11404a37b61bdddb52cd089d
I20260928 12:31:15.594077 15250 tablet_service.cc:1514] Processing DeleteTablet for tablet e514702b11404a37b61bdddb52cd089d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:15 UTC) from {username='oryx', principal='oryx/127.8.13.61@KRBTEST.COM'} at 127.0.0.1:40692
I20260928 12:31:15.594251 15523 tablet_service.cc:1514] Processing DeleteTablet for tablet e514702b11404a37b61bdddb52cd089d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:15 UTC) from {username='oryx', principal='oryx/127.8.13.61@KRBTEST.COM'} at 127.0.0.1:58182
I20260928 12:31:15.594286 15387 tablet_service.cc:1514] Processing DeleteTablet for tablet e514702b11404a37b61bdddb52cd089d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:15 UTC) from {username='oryx', principal='oryx/127.8.13.61@KRBTEST.COM'} at 127.0.0.1:48998
I20260928 12:31:15.594424 15711 tablet_replica.cc:330] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760: stopping tablet replica
I20260928 12:31:15.594457 15710 tablet_replica.cc:330] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3: stopping tablet replica
I20260928 12:31:15.594472 15711 raft_consensus.cc:2229] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:15.594511 15711 raft_consensus.cc:2258] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:15.594508 15710 raft_consensus.cc:2229] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:15.594647 15710 raft_consensus.cc:2258] T e514702b11404a37b61bdddb52cd089d P b8acc77419c94659863723a919a041f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:15.594820 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 15091
I20260928 12:31:15.594851 15711 ts_tablet_manager.cc:1905] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:15.597643 15711 ts_tablet_manager.cc:1918] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:15.597700 15711 log.cc:1196] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/e514702b11404a37b61bdddb52cd089d
I20260928 12:31:15.597980 15711 ts_tablet_manager.cc:1939] T e514702b11404a37b61bdddb52cd089d P 8b411c034e084e3a9e811cb2aceea760: Deleting consensus metadata
I20260928 12:31:15.598285 14929 catalog_manager.cc:4841] TS 8b411c034e084e3a9e811cb2aceea760 (127.8.13.3:46869): tablet e514702b11404a37b61bdddb52cd089d (table test-table [id=3ebc2f726b1f4321aeecac0e5b89a527]) successfully deleted
I20260928 12:31:15.602001 15712 tablet_replica.cc:330] T e514702b11404a37b61bdddb52cd089d P c33e75f6c55a4f39ae62d0547553141f: stopping tablet replica
I20260928 12:31:15.602144 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.602018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39495 (local address 127.0.0.1:40694)
0928 12:31:15.602076 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.602111 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:39495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 12:31:15.602216 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 15324
W20260928 12:31:15.602304 14930 catalog_manager.cc:4584] TS b8acc77419c94659863723a919a041f3 (127.8.13.1:39495): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e514702b11404a37b61bdddb52cd089d: Network error: Client connection negotiation failed: client connection to 127.8.13.1:39495: connect: Connection refused (error 111)
I20260928 12:31:15.607064 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 15460
I20260928 12:31:15.607173 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:15.607058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:44661 (local address 127.0.0.1:49016)
0928 12:31:15.607110 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:15.607137 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.2:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 12:31:15.611400 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 14810
I20260928 12:31:15.615483 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 14997
2026-09-28T12:31:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6533 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:31:15 dist-test-slave-g31s krb5kdc[15715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:31:15 dist-test-slave-g31s krb5kdc[15715](info): set up 2 sockets
Sep 28 12:31:15 dist-test-slave-g31s krb5kdc[15715](info): commencing operation
krb5kdc: starting...
W20260928 12:31:16.570010 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:31:16 dist-test-slave-g31s krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.62.keytab.
Entry for principal oryx/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.62.keytab.
I20260928 12:31:16.688928 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33295
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.62.keytab
--principal=oryx/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:33295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:16.770339 15728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:16.770555 15728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:16.770576 15728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:16.771765 15728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:16.771811 15728 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:31:16.771823 15728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:16.771835 15728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:16.771847 15728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:16.771857 15728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:16.773387 15728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:33295
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.62.keytab
--principal=oryx/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:16.773612 15728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:16.773845 15728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:16.778415 15734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:16.778439 15733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:16.778486 15728 server_base.cc:1034] running on GCE node
W20260928 12:31:16.778425 15736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:16.778843 15728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:16.779110 15728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:16.780350 15728 hybrid_clock.cc:648] HybridClock initialized: now 1790598676780280 us; error 39 us; skew 500 ppm
Sep 28 12:31:16 dist-test-slave-g31s krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598676, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:16.783350 15728 init.cc:377] Logged in from keytab as oryx/127.8.13.62@KRBTEST.COM (short username oryx)
I20260928 12:31:16.784049 15728 webserver.cc:460] Webserver started at http://127.8.13.62:41685/ using document root <none> and password file <none>
I20260928 12:31:16.784301 15728 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:16.784355 15728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:16.784466 15728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:16.785559 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "aa4838cd6de34dc5b83265ac40539ae0"
format_stamp: "Formatted at 2026-09-28 12:31:16 on dist-test-slave-g31s"
server_key: "939e691701479973f2992168d0834105"
server_key_iv: "be8d0a645f11fd20174fae42cf534d80"
server_key_version: "encryptionkey@0"
I20260928 12:31:16.785928 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "aa4838cd6de34dc5b83265ac40539ae0"
format_stamp: "Formatted at 2026-09-28 12:31:16 on dist-test-slave-g31s"
server_key: "939e691701479973f2992168d0834105"
server_key_iv: "be8d0a645f11fd20174fae42cf534d80"
server_key_version: "encryptionkey@0"
I20260928 12:31:16.787315 15728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:16.788221 15743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:16.788522 15728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:16.788604 15728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "aa4838cd6de34dc5b83265ac40539ae0"
format_stamp: "Formatted at 2026-09-28 12:31:16 on dist-test-slave-g31s"
server_key: "939e691701479973f2992168d0834105"
server_key_iv: "be8d0a645f11fd20174fae42cf534d80"
server_key_version: "encryptionkey@0"
I20260928 12:31:16.788681 15728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:16.797341 15728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:16.799328 15728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:16.799458 15728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:16.803974 15728 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:33295
I20260928 12:31:16.804021 15805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:33295 every 8 connection(s)
I20260928 12:31:16.804348 15728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:16.804965 15806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:16.807256 15806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0: Bootstrap starting.
I20260928 12:31:16.807826 15806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:16.808058 15806 log.cc:824] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:16.808718 15806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0: No bootstrap required, opened a new log
I20260928 12:31:16.810815 15806 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4838cd6de34dc5b83265ac40539ae0" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33295 } }
I20260928 12:31:16.810961 15806 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:16.810981 15806 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa4838cd6de34dc5b83265ac40539ae0, State: Initialized, Role: FOLLOWER
I20260928 12:31:16.811092 15806 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4838cd6de34dc5b83265ac40539ae0" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33295 } }
I20260928 12:31:16.811163 15806 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:16.811185 15806 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:16.811218 15806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:16.811715 15806 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4838cd6de34dc5b83265ac40539ae0" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33295 } }
I20260928 12:31:16.811800 15806 leader_election.cc:302] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa4838cd6de34dc5b83265ac40539ae0; no voters:
I20260928 12:31:16.811921 15806 leader_election.cc:288] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:16.811996 15809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:16.812150 15806 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:16.812160 15809 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [term 1 LEADER]: Becoming Leader. State: Replica: aa4838cd6de34dc5b83265ac40539ae0, State: Running, Role: LEADER
I20260928 12:31:16.812316 15809 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4838cd6de34dc5b83265ac40539ae0" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33295 } }
I20260928 12:31:16.812654 15811 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa4838cd6de34dc5b83265ac40539ae0. Latest consensus state: current_term: 1 leader_uuid: "aa4838cd6de34dc5b83265ac40539ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4838cd6de34dc5b83265ac40539ae0" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33295 } } }
I20260928 12:31:16.812849 15810 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa4838cd6de34dc5b83265ac40539ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4838cd6de34dc5b83265ac40539ae0" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33295 } } }
I20260928 12:31:16.812911 15811 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:16.812970 15810 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:16.813256 15818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:16.813684 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 15728
I20260928 12:31:16.813783 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:16.813973 8244 external_mini_cluster.cc:1428] Setting key b9b4433d2b6db359d8b30b42faa96b2f
I20260928 12:31:16.814527 15818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:16.816022 15818 catalog_manager.cc:1347] Generated new cluster ID: 462d60f0df714b528bcc82e9b167fbb8
I20260928 12:31:16.816075 15818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:31:16 dist-test-slave-g31s krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
I20260928 12:31:16.823040 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:16.814442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.8.13.62:33295)
0928 12:31:16.814673 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:16.814681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:16.814711 (+ 30us) server_negotiation.cc:407] Connection header received
0928 12:31:16.814829 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:16.814836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:16.814906 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:16.815036 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:16.815699 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:16.816700 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:16.817437 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:16.817916 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:16.818007 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:16.820136 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:16.820140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:16.820141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:16.820154 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:16.821640 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:16.822020 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:16.822023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:16.822023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:16.822106 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:16.822271 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:16.822272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:16.822272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:16.822512 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:16.822659 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:16.822729 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 12:31:16.822866 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260928 12:31:16.846202 15818 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:16.846957 15818 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:16.861202 15818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0: Generated new TSK 0
I20260928 12:31:16.861445 15818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab.
Entry for principal oryx/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab.
I20260928 12:31:16.890691 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33295
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab
--principal=oryx/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:16.974493 15832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:16.974699 15832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:16.974725 15832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:16.975973 15832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:16.976017 15832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:16.976039 15832 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:31:16.976058 15832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:16.977615 15832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab
--principal=oryx/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.62:33295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:16.977861 15832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:16.978070 15832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:16.983106 15840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:16.983096 15832 server_base.cc:1034] running on GCE node
W20260928 12:31:16.983088 15837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:16.983084 15838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:16.983577 15832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:16.983901 15832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:16.985141 15832 hybrid_clock.cc:648] HybridClock initialized: now 1790598676985109 us; error 40 us; skew 500 ppm
Sep 28 12:31:16 dist-test-slave-g31s krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598676, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:16.987960 15832 init.cc:377] Logged in from keytab as oryx/127.8.13.1@KRBTEST.COM (short username oryx)
I20260928 12:31:16.988420 15832 webserver.cc:460] Webserver started at http://127.8.13.1:45845/ using document root <none> and password file <none>
I20260928 12:31:16.988627 15832 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:16.988667 15832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:16.988771 15832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:16.989845 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "f1f741011ece4b2eba023c30586e321a"
format_stamp: "Formatted at 2026-09-28 12:31:16 on dist-test-slave-g31s"
server_key: "c67b2a3768f41cf653f7c60b8295782c"
server_key_iv: "23fa0aa9fcd3ca2ec1c884226eeaa894"
server_key_version: "encryptionkey@0"
I20260928 12:31:16.990176 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "f1f741011ece4b2eba023c30586e321a"
format_stamp: "Formatted at 2026-09-28 12:31:16 on dist-test-slave-g31s"
server_key: "c67b2a3768f41cf653f7c60b8295782c"
server_key_iv: "23fa0aa9fcd3ca2ec1c884226eeaa894"
server_key_version: "encryptionkey@0"
I20260928 12:31:16.991515 15832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:16.992281 15847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:16.992453 15832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:16.992551 15832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "f1f741011ece4b2eba023c30586e321a"
format_stamp: "Formatted at 2026-09-28 12:31:16 on dist-test-slave-g31s"
server_key: "c67b2a3768f41cf653f7c60b8295782c"
server_key_iv: "23fa0aa9fcd3ca2ec1c884226eeaa894"
server_key_version: "encryptionkey@0"
I20260928 12:31:16.992628 15832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:17.009967 15832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:17.011821 15832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:17.011929 15832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:17.012420 15832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:17.012462 15832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:17.012485 15832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:17.012499 15832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:17.018657 15832 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:33627
I20260928 12:31:17.018806 15962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:33627 every 8 connection(s)
I20260928 12:31:17.019071 15832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:31:17 dist-test-slave-g31s krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598676, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
I20260928 12:31:17.022648 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.012897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:41883 (local address 127.8.13.62:33295)
0928 12:31:17.012935 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.012937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.013175 (+ 238us) server_negotiation.cc:407] Connection header received
0928 12:31:17.013792 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.013810 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.013824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.013901 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:17.014724 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.015514 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.016994 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.017322 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.017402 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.020030 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:17.020031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:17.020032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:17.020043 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:17.021336 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.021699 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.021700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.021701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.021757 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.022000 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.022001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.022002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.022117 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:17.022172 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.022446 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.022555 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:31:17.022778 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.012911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33295 (local address 127.8.13.1:41883)
0928 12:31:17.013047 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.013081 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.013619 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.013995 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.013999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.014345 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:17.014620 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.014625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.015605 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.015606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.016903 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.016906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.017442 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.017443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.017613 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.018232 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:17.018241 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:17.019935 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:17.021414 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.021421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.021423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.021631 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.021841 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.021842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.021842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.021945 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.022170 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:17.022176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:17.022322 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.022411 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.022569 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260928 12:31:17.023290 15963 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33295
I20260928 12:31:17.023378 15963 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:17.023587 15963 heartbeater.cc:511] Master 127.8.13.62:33295 requested a full tablet report, sending...
I20260928 12:31:17.024144 15759 ts_manager.cc:194] Registered new tserver with Master: f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627)
I20260928 12:31:17.024957 15759 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:41883
I20260928 12:31:17.025472 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 15832
I20260928 12:31:17.025595 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:31:17.025830 8244 external_mini_cluster.cc:1428] Setting key ec51001d42de36dc79ddec21a8bf5206
I20260928 12:31:17.029830 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.024462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:47907 (local address 127.8.13.62:33295)
0928 12:31:17.024499 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.024501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.024510 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:17.024529 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.024529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.024539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.024591 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:17.024845 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.025470 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.026272 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.026473 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.026528 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.027369 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:17.027370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:17.027370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:17.027377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:17.028670 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.029020 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.029021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.029021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.029077 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.029288 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.029288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.029288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.029422 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:17.029485 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.029612 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.029701 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:31:17.029827 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.024274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33295 (local address 127.8.13.1:47907)
0928 12:31:17.024405 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.024415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.024469 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.024594 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.024594 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.024741 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:17.024792 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.024794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.025558 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.025559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.026199 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.026201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.026531 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.026532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.026591 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.026992 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:17.026997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:17.027280 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:17.028772 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.028773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.028773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.028948 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.029159 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.029160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.029160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.029226 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.029478 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:17.029479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:17.029520 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.029567 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.029680 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab.
Entry for principal oryx/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab.
I20260928 12:31:17.050747 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33295
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab
--principal=oryx/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:17.130390 15972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:17.130620 15972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:17.130648 15972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:17.131897 15972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:17.131943 15972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:17.131965 15972 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:31:17.131982 15972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:17.133534 15972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab
--principal=oryx/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.62:33295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:17.133750 15972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:17.133965 15972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:17.138792 15978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:17.138898 15972 server_base.cc:1034] running on GCE node
W20260928 12:31:17.138775 15977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:17.138789 15980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:17.139194 15972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:17.139408 15972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:17.140560 15972 hybrid_clock.cc:648] HybridClock initialized: now 1790598677140546 us; error 26 us; skew 500 ppm
Sep 28 12:31:17 dist-test-slave-g31s krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598677, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:17.143256 15972 init.cc:377] Logged in from keytab as oryx/127.8.13.2@KRBTEST.COM (short username oryx)
I20260928 12:31:17.143751 15972 webserver.cc:460] Webserver started at http://127.8.13.2:32789/ using document root <none> and password file <none>
I20260928 12:31:17.143954 15972 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:17.143998 15972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:17.144106 15972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:17.145138 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "0f5fedee945c49bbb2d564ade40d626b"
format_stamp: "Formatted at 2026-09-28 12:31:17 on dist-test-slave-g31s"
server_key: "214c969b6046b1d6ba70fdc32062f3ca"
server_key_iv: "0944547f6712c1e7eb0e507bda6246ca"
server_key_version: "encryptionkey@0"
I20260928 12:31:17.145462 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "0f5fedee945c49bbb2d564ade40d626b"
format_stamp: "Formatted at 2026-09-28 12:31:17 on dist-test-slave-g31s"
server_key: "214c969b6046b1d6ba70fdc32062f3ca"
server_key_iv: "0944547f6712c1e7eb0e507bda6246ca"
server_key_version: "encryptionkey@0"
I20260928 12:31:17.146739 15972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:31:17.147424 15987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:17.147564 15972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:31:17.147643 15972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "0f5fedee945c49bbb2d564ade40d626b"
format_stamp: "Formatted at 2026-09-28 12:31:17 on dist-test-slave-g31s"
server_key: "214c969b6046b1d6ba70fdc32062f3ca"
server_key_iv: "0944547f6712c1e7eb0e507bda6246ca"
server_key_version: "encryptionkey@0"
I20260928 12:31:17.147702 15972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:17.157433 15972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:17.160245 15972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:17.160418 15972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:17.161021 15972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:17.161078 15972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:17.161120 15972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:17.161137 15972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:31:17 dist-test-slave-g31s krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598677, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
I20260928 12:31:17.168141 15972 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:42057
I20260928 12:31:17.168653 15972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:31:17.169623 16102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:42057 every 8 connection(s)
I20260928 12:31:17.171267 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.161597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:42455 (local address 127.8.13.62:33295)
0928 12:31:17.161685 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.161687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.161920 (+ 233us) server_negotiation.cc:407] Connection header received
0928 12:31:17.162558 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.162559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.162577 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.162679 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:17.163369 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.164132 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.165193 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.165388 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.165445 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.168539 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:17.168541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:17.168541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:17.168550 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:17.170104 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.170502 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.170503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.170503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.170550 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.170709 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.170709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.170709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.170821 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:17.170867 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.171126 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.171200 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 12:31:17.171414 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.161522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33295 (local address 127.8.13.2:42455)
0928 12:31:17.161742 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.161792 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.162440 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.162764 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.162768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.163049 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:17.163258 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.163267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.164235 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.164237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.165082 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.165086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.165484 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.165485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.165630 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.166249 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:17.166259 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:17.168257 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:17.170193 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.170199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.170201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.170437 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.170585 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.170585 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.170586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.170669 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.170866 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:17.170868 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:17.170986 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.171068 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.171197 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":121,"threads_started":1}
I20260928 12:31:17.171859 16103 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33295
I20260928 12:31:17.171938 16103 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:17.172106 16103 heartbeater.cc:511] Master 127.8.13.62:33295 requested a full tablet report, sending...
I20260928 12:31:17.172477 15760 ts_manager.cc:194] Registered new tserver with Master: 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2:42057)
I20260928 12:31:17.173003 15760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:42455
I20260928 12:31:17.175436 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 15972
I20260928 12:31:17.175549 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:31:17.175753 8244 external_mini_cluster.cc:1428] Setting key 0b66bcb14a6c9bfc905ad7e90a48d9e0
I20260928 12:31:17.178638 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.172842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:45129 (local address 127.8.13.62:33295)
0928 12:31:17.172879 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.172881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.172950 (+ 69us) server_negotiation.cc:407] Connection header received
0928 12:31:17.172998 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.172999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.173008 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.173072 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:17.173422 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.173940 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.174642 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.174884 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.174980 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.176107 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:17.176108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:17.176109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:17.176116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:17.177425 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.177832 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.177833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.177833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.177896 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.178094 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.178095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.178095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.178245 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:17.178304 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.178461 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.178543 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:31:17.178651 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.172781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33295 (local address 127.8.13.2:45129)
0928 12:31:17.172898 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.172909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.172960 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.173079 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.173080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.173257 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:17.173344 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.173347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.174016 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.174017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.174506 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.174507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.175010 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.175012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.175074 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.175567 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:17.175572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:17.175961 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:17.177499 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.177501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.177501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.177709 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.177967 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.177968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.177969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.178042 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.178324 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:17.178325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:17.178371 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.178431 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.178543 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
WARNING: no policy specified for oryx/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab.
Entry for principal oryx/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab.
I20260928 12:31:17.200981 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33295
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab
--principal=oryx/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:17.280347 16112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:17.280593 16112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:17.280618 16112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:17.281898 16112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:17.281952 16112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:17.281983 16112 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:31:17.282007 16112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:17.283696 16112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab
--principal=oryx/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.62:33295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:17.283952 16112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:17.284178 16112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:17.288945 16118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:17.288967 16120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:17.288954 16117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:17.289100 16112 server_base.cc:1034] running on GCE node
I20260928 12:31:17.289325 16112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:17.289569 16112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:17.290724 16112 hybrid_clock.cc:648] HybridClock initialized: now 1790598677290711 us; error 35 us; skew 500 ppm
Sep 28 12:31:17 dist-test-slave-g31s krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598677, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:17.293164 16112 init.cc:377] Logged in from keytab as oryx/127.8.13.3@KRBTEST.COM (short username oryx)
I20260928 12:31:17.293586 16112 webserver.cc:460] Webserver started at http://127.8.13.3:39485/ using document root <none> and password file <none>
I20260928 12:31:17.293787 16112 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:17.293874 16112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:17.294015 16112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:17.295131 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "2ae1f2585b8644b283c4ed0d1a569514"
format_stamp: "Formatted at 2026-09-28 12:31:17 on dist-test-slave-g31s"
server_key: "a1232f54188fbb95c5fcf1968c97d07b"
server_key_iv: "3d61876a639aa2c3b14d17ba70becd7f"
server_key_version: "encryptionkey@0"
I20260928 12:31:17.295455 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "2ae1f2585b8644b283c4ed0d1a569514"
format_stamp: "Formatted at 2026-09-28 12:31:17 on dist-test-slave-g31s"
server_key: "a1232f54188fbb95c5fcf1968c97d07b"
server_key_iv: "3d61876a639aa2c3b14d17ba70becd7f"
server_key_version: "encryptionkey@0"
I20260928 12:31:17.296753 16112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:31:17.297415 16127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:17.297569 16112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:31:17.297652 16112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "2ae1f2585b8644b283c4ed0d1a569514"
format_stamp: "Formatted at 2026-09-28 12:31:17 on dist-test-slave-g31s"
server_key: "a1232f54188fbb95c5fcf1968c97d07b"
server_key_iv: "3d61876a639aa2c3b14d17ba70becd7f"
server_key_version: "encryptionkey@0"
I20260928 12:31:17.297709 16112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:17.317401 16112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:17.319422 16112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:17.319562 16112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:17.320065 16112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:17.320106 16112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:17.320137 16112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:17.320163 16112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:31:17 dist-test-slave-g31s krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598677, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
I20260928 12:31:17.327538 16112 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:42117
I20260928 12:31:17.327782 16242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:42117 every 8 connection(s)
I20260928 12:31:17.327970 16112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:31:17.330627 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.320561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:37805 (local address 127.8.13.62:33295)
0928 12:31:17.320620 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.320622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.320873 (+ 251us) server_negotiation.cc:407] Connection header received
0928 12:31:17.321547 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.321548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.321563 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.321637 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:17.322633 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.323343 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.324880 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.325175 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.325244 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.327958 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:17.327959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:17.327959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:17.327967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:17.329197 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.329657 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.329658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.329658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.329715 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.329976 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.329977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.329977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.330087 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:17.330138 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.330483 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.330560 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:31:17.330714 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.320503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33295 (local address 127.8.13.3:37805)
0928 12:31:17.320683 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.320732 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.321419 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.321724 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.321729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.322143 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:17.322516 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.322524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.323418 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.323420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.324766 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.324770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.325267 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.325268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.325429 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.326036 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:17.326044 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:17.327846 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:17.329333 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.329339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.329341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.329568 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.329777 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.329777 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.329778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.329924 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.330164 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:17.330170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:17.330312 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.330398 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.330531 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260928 12:31:17.331176 16243 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33295
I20260928 12:31:17.331271 16243 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:17.331452 16243 heartbeater.cc:511] Master 127.8.13.62:33295 requested a full tablet report, sending...
I20260928 12:31:17.331813 15759 ts_manager.cc:194] Registered new tserver with Master: 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117)
I20260928 12:31:17.332175 15759 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:37805
I20260928 12:31:17.335762 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 16112
I20260928 12:31:17.335879 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:31:17.336067 8244 external_mini_cluster.cc:1428] Setting key 8b09057e32a591bfefd6dbbca6bdfa51
I20260928 12:31:17.337152 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:31:17.338413 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.332456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:44557 (local address 127.8.13.62:33295)
0928 12:31:17.332495 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.332497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.333030 (+ 533us) server_negotiation.cc:407] Connection header received
0928 12:31:17.333071 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.333072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.333082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.333126 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:17.333429 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.333957 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.334615 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.334812 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.334865 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.335781 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:17.335783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:17.335783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:17.335791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:17.337244 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.337561 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.337562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.337562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.337627 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.337895 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.337896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.337896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.338022 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:17.338088 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.338255 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.338332 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:31:17.338420 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.332420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33295 (local address 127.8.13.3:44557)
0928 12:31:17.332956 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.332966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.333022 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.333128 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.333129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.333306 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:17.333363 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.333366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.334037 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.334038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.334542 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.334544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.334878 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.334878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.334942 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.335399 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:17.335404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:17.335695 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:17.337323 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.337325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.337325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.337500 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.337733 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.337734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.337734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.337832 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.338111 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:17.338112 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:17.338159 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.338222 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.338317 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":69,"threads_started":1}
Sep 28 12:31:17 dist-test-slave-g31s krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598677, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:31:17 dist-test-slave-g31s krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:17 dist-test-slave-g31s krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:31:17.359647 15828 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:17.354273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.8.13.62:33295)
0928 12:31:17.354408 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.354410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.354420 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:17.354446 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.354447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.354459 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.354514 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:17.355089 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.355586 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.356357 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.356546 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.356606 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.359559 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39418: BlockingRecv error: recv got EOF from 127.0.0.1:39418 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 28 12:31:17 dist-test-slave-g31s krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598677, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
I20260928 12:31:17.367461 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.360580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.8.13.62:33295)
0928 12:31:17.360629 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.360630 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.360641 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:31:17.360698 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.360699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.360708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.360760 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:17.361311 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.361893 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.362586 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.362786 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.362853 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.364990 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:17.364992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:17.364992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:17.365000 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:17.366260 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.366665 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.366666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.366667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.366723 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.366943 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.366944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.366945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.367081 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:17.367166 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.367304 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.367374 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:31:17.369233 15759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:31:17.370922 15759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:31:17.377785 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.373485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:42057 (local address 127.0.0.1:54516)
0928 12:31:17.373644 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.373689 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.373764 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.374132 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.374136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.374139 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:17.374248 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.374251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.375471 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.375472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.376472 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.376475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.377436 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.377438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.377535 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.377553 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.377631 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.377692 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260928 12:31:17.377839 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.373560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.8.13.2:42057)
0928 12:31:17.373895 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.373902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.373918 (+ 16us) server_negotiation.cc:407] Connection header received
0928 12:31:17.373979 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.373983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.374005 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.374102 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:17.374335 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.375364 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.376555 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.377349 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.377429 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.377515 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.377652 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.377699 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":106,"threads_started":1}
I20260928 12:31:17.378053 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.373719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:42117 (local address 127.0.0.1:50494)
0928 12:31:17.374007 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.374019 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.374070 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.374605 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.374606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.374609 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:17.374765 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.374769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.376135 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.376136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.376992 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.376994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.377786 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.377786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.377907 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.377911 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.377960 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.377998 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":32,"thread_start_us":131,"threads_started":1}
I20260928 12:31:17.378289 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.374055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.8.13.3:42117)
0928 12:31:17.374429 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.374434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.374446 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:31:17.374485 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.374487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.374505 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.374599 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:17.374897 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.376039 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.377075 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.377630 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.377686 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.377753 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.378150 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.378216 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":91,"threads_started":1}
I20260928 12:31:17.378857 16037 tablet_service.cc:1467] Processing CreateTablet for tablet cd3afcb12f3f42a081a88bde60ebb65a (DEFAULT_TABLE table=test-table [id=5f3477ee6d5441a1a010764315095d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:17.378896 16172 tablet_service.cc:1467] Processing CreateTablet for tablet cd3afcb12f3f42a081a88bde60ebb65a (DEFAULT_TABLE table=test-table [id=5f3477ee6d5441a1a010764315095d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:17.379195 16037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3afcb12f3f42a081a88bde60ebb65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:17.379206 16172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3afcb12f3f42a081a88bde60ebb65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:17.382448 16271 tablet_bootstrap.cc:492] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: Bootstrap starting.
I20260928 12:31:17.383117 16271 tablet_bootstrap.cc:654] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:17.383265 16272 tablet_bootstrap.cc:492] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514: Bootstrap starting.
I20260928 12:31:17.383450 16271 log.cc:824] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:17.383606 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.373944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:33627 (local address 127.0.0.1:41146)
0928 12:31:17.374739 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.374747 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.374795 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.374994 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.374995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.374997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:17.375036 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.375038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.376274 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.376274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.377382 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.377385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.383332 (+ 5947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.383334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.383411 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.383416 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.383473 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.383527 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":20,"thread_start_us":81,"threads_started":1}
I20260928 12:31:17.383872 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.374242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.8.13.1:33627)
0928 12:31:17.374694 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.374697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.374810 (+ 113us) server_negotiation.cc:407] Connection header received
0928 12:31:17.374858 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.374860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.374874 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.374935 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:17.375126 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.376166 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.377474 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.383115 (+ 5641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.383172 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.383257 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.383495 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.383626 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":75,"threads_started":1}
I20260928 12:31:17.383919 16272 tablet_bootstrap.cc:654] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:17.384445 16271 tablet_bootstrap.cc:492] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: No bootstrap required, opened a new log
I20260928 12:31:17.384506 16271 ts_tablet_manager.cc:1397] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:31:17.384616 16272 log.cc:824] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:17.384545 15897 tablet_service.cc:1467] Processing CreateTablet for tablet cd3afcb12f3f42a081a88bde60ebb65a (DEFAULT_TABLE table=test-table [id=5f3477ee6d5441a1a010764315095d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:17.384891 15897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3afcb12f3f42a081a88bde60ebb65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:17.385455 16272 tablet_bootstrap.cc:492] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514: No bootstrap required, opened a new log
I20260928 12:31:17.385524 16272 ts_tablet_manager.cc:1397] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:31:17.386317 16271 raft_consensus.cc:348] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } }
I20260928 12:31:17.386453 16271 raft_consensus.cc:374] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:17.386483 16271 raft_consensus.cc:729] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5fedee945c49bbb2d564ade40d626b, State: Initialized, Role: FOLLOWER
I20260928 12:31:17.386620 16271 consensus_queue.cc:260] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } }
I20260928 12:31:17.386881 16271 ts_tablet_manager.cc:1428] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 12:31:17.386914 16103 heartbeater.cc:503] Master 127.8.13.62:33295 was elected leader, sending a full tablet report...
I20260928 12:31:17.387889 16272 raft_consensus.cc:348] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } }
I20260928 12:31:17.388056 16272 raft_consensus.cc:374] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:17.388103 16272 raft_consensus.cc:729] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae1f2585b8644b283c4ed0d1a569514, State: Initialized, Role: FOLLOWER
I20260928 12:31:17.388216 16272 consensus_queue.cc:260] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } }
I20260928 12:31:17.388502 16243 heartbeater.cc:503] Master 127.8.13.62:33295 was elected leader, sending a full tablet report...
I20260928 12:31:17.388535 16277 tablet_bootstrap.cc:492] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: Bootstrap starting.
I20260928 12:31:17.388710 16272 ts_tablet_manager.cc:1428] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:31:17.389169 16277 tablet_bootstrap.cc:654] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:17.389583 16277 log.cc:824] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:17.390497 16277 tablet_bootstrap.cc:492] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: No bootstrap required, opened a new log
I20260928 12:31:17.390563 16277 ts_tablet_manager.cc:1397] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:31:17.392045 16277 raft_consensus.cc:348] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } }
I20260928 12:31:17.392170 16277 raft_consensus.cc:374] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:17.392190 16277 raft_consensus.cc:729] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f741011ece4b2eba023c30586e321a, State: Initialized, Role: FOLLOWER
I20260928 12:31:17.392300 16277 consensus_queue.cc:260] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } }
I20260928 12:31:17.392575 16277 ts_tablet_manager.cc:1428] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:31:17.392622 15963 heartbeater.cc:503] Master 127.8.13.62:33295 was elected leader, sending a full tablet report...
W20260928 12:31:17.419757 16104 tablet.cc:2369] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:31:17.503813 16275 raft_consensus.cc:482] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:17.503952 16275 raft_consensus.cc:504] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } }
I20260928 12:31:17.504282 16275 leader_election.cc:288] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117), f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627)
I20260928 12:31:17.507922 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.504643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:43553 (local address 127.8.13.1:33627)
0928 12:31:17.504685 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.504687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.504811 (+ 124us) server_negotiation.cc:407] Connection header received
0928 12:31:17.504857 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.504858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.504870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.504913 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:17.505051 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.505535 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.506970 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.507532 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.507615 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.507628 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.507808 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.507849 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:31:17.507939 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.504572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:57111 (local address 127.8.13.3:42117)
0928 12:31:17.504618 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.504620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.504630 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:17.504704 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.504705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.504722 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.504769 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:17.505202 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.505773 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.507077 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.507630 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.507692 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.507700 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.507843 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.507887 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:31:17.507959 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.504468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:42117 (local address 127.8.13.2:57111)
0928 12:31:17.504528 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.504544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.504614 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.504780 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.504781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.504782 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:17.504849 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.504851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.505914 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.505915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.506955 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.506957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.507718 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.507719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.507780 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.507783 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.507829 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.507863 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:31:17.507967 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.504563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:33627 (local address 127.8.13.2:43553)
0928 12:31:17.504728 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.504746 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.504804 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.504922 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.504923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.504924 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:17.504989 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.504991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.505613 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.505614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.506838 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.506842 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.507630 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.507631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.507720 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.507725 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.507797 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.507856 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":13,"thread_start_us":107,"threads_started":1}
I20260928 12:31:17.508368 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3afcb12f3f42a081a88bde60ebb65a" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f741011ece4b2eba023c30586e321a" is_pre_election: true
I20260928 12:31:17.508427 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3afcb12f3f42a081a88bde60ebb65a" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae1f2585b8644b283c4ed0d1a569514" is_pre_election: true
I20260928 12:31:17.508512 15917 raft_consensus.cc:2454] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 0.
I20260928 12:31:17.508579 16197 raft_consensus.cc:2454] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 0.
I20260928 12:31:17.508726 15991 leader_election.cc:302] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5fedee945c49bbb2d564ade40d626b, f1f741011ece4b2eba023c30586e321a; no voters:
I20260928 12:31:17.508906 16275 raft_consensus.cc:2781] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:31:17.508973 16275 raft_consensus.cc:482] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:31:17.508988 16275 raft_consensus.cc:3037] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:17.509976 16275 raft_consensus.cc:504] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } }
I20260928 12:31:17.510092 16275 leader_election.cc:288] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 election: Requested vote from peers 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117), f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627)
I20260928 12:31:17.510293 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3afcb12f3f42a081a88bde60ebb65a" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae1f2585b8644b283c4ed0d1a569514"
I20260928 12:31:17.510311 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3afcb12f3f42a081a88bde60ebb65a" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f741011ece4b2eba023c30586e321a"
I20260928 12:31:17.510370 16197 raft_consensus.cc:3037] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:17.510390 15917 raft_consensus.cc:3037] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:17.511029 16197 raft_consensus.cc:2454] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 1.
I20260928 12:31:17.511142 15917 raft_consensus.cc:2454] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 1.
I20260928 12:31:17.511188 15991 leader_election.cc:302] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5fedee945c49bbb2d564ade40d626b, 2ae1f2585b8644b283c4ed0d1a569514; no voters:
I20260928 12:31:17.511261 16275 raft_consensus.cc:2781] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:17.511413 16275 raft_consensus.cc:686] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 1 LEADER]: Becoming Leader. State: Replica: 0f5fedee945c49bbb2d564ade40d626b, State: Running, Role: LEADER
I20260928 12:31:17.511498 16275 consensus_queue.cc:237] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } }
I20260928 12:31:17.512287 15759 catalog_manager.cc:5495] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2). New cstate: current_term: 1 leader_uuid: "0f5fedee945c49bbb2d564ade40d626b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } health_report { overall_health: UNKNOWN } } }
W20260928 12:31:17.519845 15964 tablet.cc:2369] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:31:17.522887 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.520570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.8.13.2:42057)
0928 12:31:17.520610 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.520612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.520620 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:17.520657 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.520658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.520680 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.520750 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:31:17.520885 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.521355 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.522147 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.522350 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.522402 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.522552 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:31:17.522655 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:31:17.522722 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.522755 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.522777 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:31:17.524601 16197 raft_consensus.cc:1261] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Refusing update from remote peer 0f5fedee945c49bbb2d564ade40d626b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:17.524601 15917 raft_consensus.cc:1261] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Refusing update from remote peer 0f5fedee945c49bbb2d564ade40d626b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:17.524897 16281 consensus_queue.cc:1035] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:17.525028 16275 consensus_queue.cc:1035] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:17.528168 16286 mvcc.cc:205] Tried to move back new op lower bound from 7334292183138254848 to 7334292183087636480. Current Snapshot: MvccSnapshot[applied={T|T < 7334292183138254848}]
I20260928 12:31:17.529070 16285 mvcc.cc:205] Tried to move back new op lower bound from 7334292183138254848 to 7334292183087636480. Current Snapshot: MvccSnapshot[applied={T|T < 7334292183138254848 or (T in {7334292183138254848})}]
I20260928 12:31:17.530040 15759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:17.530129 15759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:17.530819 15759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0: Sending DeleteTablet for 3 replicas of tablet cd3afcb12f3f42a081a88bde60ebb65a
I20260928 12:31:17.531124 16172 tablet_service.cc:1514] Processing DeleteTablet for tablet cd3afcb12f3f42a081a88bde60ebb65a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:17 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:50494
I20260928 12:31:17.531172 15897 tablet_service.cc:1514] Processing DeleteTablet for tablet cd3afcb12f3f42a081a88bde60ebb65a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:17 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:41146
I20260928 12:31:17.531172 16037 tablet_service.cc:1514] Processing DeleteTablet for tablet cd3afcb12f3f42a081a88bde60ebb65a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:17 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:54516
I20260928 12:31:17.531348 16295 tablet_replica.cc:330] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: stopping tablet replica
I20260928 12:31:17.531350 16294 tablet_replica.cc:330] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514: stopping tablet replica
I20260928 12:31:17.531437 16295 raft_consensus.cc:2229] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:17.531440 16294 raft_consensus.cc:2229] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:17.531531 16295 raft_consensus.cc:2258] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:17.531762 16296 tablet_replica.cc:330] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: stopping tablet replica
I20260928 12:31:17.531778 16294 raft_consensus.cc:2258] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:17.531850 16296 raft_consensus.cc:2229] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:31:17.531808 15759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 12:31:17.531930 16296 raft_consensus.cc:2258] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 12:31:17.531937 15759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:31:17.532434 16296 ts_tablet_manager.cc:1905] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:17.532805 16294 ts_tablet_manager.cc:1905] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:17.533131 16295 ts_tablet_manager.cc:1905] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:17.533394 15897 tablet_service.cc:1467] Processing CreateTablet for tablet 8dcc5ad550e5424cb176bca9cf479466 (DEFAULT_TABLE table=test-table [id=c15a5f03e0b244738cc5d37a7697ba50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:17.533524 15897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dcc5ad550e5424cb176bca9cf479466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:17.534194 16296 ts_tablet_manager.cc:1918] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:17.534267 16296 log.cc:1196] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/cd3afcb12f3f42a081a88bde60ebb65a
I20260928 12:31:17.534502 16296 ts_tablet_manager.cc:1939] T cd3afcb12f3f42a081a88bde60ebb65a P 0f5fedee945c49bbb2d564ade40d626b: Deleting consensus metadata
I20260928 12:31:17.535006 16277 tablet_bootstrap.cc:492] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a: Bootstrap starting.
I20260928 12:31:17.535013 16295 ts_tablet_manager.cc:1918] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:17.535125 16295 log.cc:1196] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/wal/wals/cd3afcb12f3f42a081a88bde60ebb65a
I20260928 12:31:17.535355 16295 ts_tablet_manager.cc:1939] T cd3afcb12f3f42a081a88bde60ebb65a P f1f741011ece4b2eba023c30586e321a: Deleting consensus metadata
I20260928 12:31:17.535414 16277 tablet_bootstrap.cc:654] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:17.535712 15747 catalog_manager.cc:4841] TS f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627): tablet cd3afcb12f3f42a081a88bde60ebb65a (table test-table [id=5f3477ee6d5441a1a010764315095d40]) successfully deleted
I20260928 12:31:17.536098 15747 catalog_manager.cc:4841] TS 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2:42057): tablet cd3afcb12f3f42a081a88bde60ebb65a (table test-table [id=5f3477ee6d5441a1a010764315095d40]) successfully deleted
I20260928 12:31:17.534924 16172 tablet_service.cc:1467] Processing CreateTablet for tablet 8dcc5ad550e5424cb176bca9cf479466 (DEFAULT_TABLE table=test-table [id=c15a5f03e0b244738cc5d37a7697ba50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:17.536240 16172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dcc5ad550e5424cb176bca9cf479466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:17.536971 16294 ts_tablet_manager.cc:1918] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:17.537045 16294 log.cc:1196] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/cd3afcb12f3f42a081a88bde60ebb65a
I20260928 12:31:17.537249 16294 ts_tablet_manager.cc:1939] T cd3afcb12f3f42a081a88bde60ebb65a P 2ae1f2585b8644b283c4ed0d1a569514: Deleting consensus metadata
I20260928 12:31:17.537557 16272 tablet_bootstrap.cc:492] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514: Bootstrap starting.
I20260928 12:31:17.537950 15747 catalog_manager.cc:4841] TS 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117): tablet cd3afcb12f3f42a081a88bde60ebb65a (table test-table [id=5f3477ee6d5441a1a010764315095d40]) successfully deleted
I20260928 12:31:17.537992 16272 tablet_bootstrap.cc:654] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:17.538023 16037 tablet_service.cc:1467] Processing CreateTablet for tablet 8dcc5ad550e5424cb176bca9cf479466 (DEFAULT_TABLE table=test-table [id=c15a5f03e0b244738cc5d37a7697ba50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:17.538133 16037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dcc5ad550e5424cb176bca9cf479466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:17.538590 16272 tablet_bootstrap.cc:492] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514: No bootstrap required, opened a new log
I20260928 12:31:17.538630 16272 ts_tablet_manager.cc:1397] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:17.538770 16272 raft_consensus.cc:348] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.538839 16272 raft_consensus.cc:374] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:17.538857 16272 raft_consensus.cc:729] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae1f2585b8644b283c4ed0d1a569514, State: Initialized, Role: FOLLOWER
I20260928 12:31:17.538918 16272 consensus_queue.cc:260] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.539005 16272 ts_tablet_manager.cc:1428] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:17.539114 16271 tablet_bootstrap.cc:492] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b: Bootstrap starting.
I20260928 12:31:17.539501 16271 tablet_bootstrap.cc:654] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:17.539896 16277 tablet_bootstrap.cc:492] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a: No bootstrap required, opened a new log
I20260928 12:31:17.539945 16277 ts_tablet_manager.cc:1397] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 12:31:17.540086 16277 raft_consensus.cc:348] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.540143 16277 raft_consensus.cc:374] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:17.540156 16277 raft_consensus.cc:729] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f741011ece4b2eba023c30586e321a, State: Initialized, Role: FOLLOWER
I20260928 12:31:17.540199 16277 consensus_queue.cc:260] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.540304 16277 ts_tablet_manager.cc:1428] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:31:17.540710 16271 tablet_bootstrap.cc:492] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b: No bootstrap required, opened a new log
I20260928 12:31:17.540755 16271 ts_tablet_manager.cc:1397] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:31:17.540884 16271 raft_consensus.cc:348] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.540944 16271 raft_consensus.cc:374] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:17.540957 16271 raft_consensus.cc:729] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5fedee945c49bbb2d564ade40d626b, State: Initialized, Role: FOLLOWER
I20260928 12:31:17.540987 16271 consensus_queue.cc:260] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.541064 16271 ts_tablet_manager.cc:1428] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260928 12:31:17.578797 16244 tablet.cc:2369] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:31:17.771747 16276 raft_consensus.cc:482] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:17.771845 16276 raft_consensus.cc:504] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.772178 16276 leader_election.cc:288] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627), 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2:42057)
I20260928 12:31:17.772610 16289 raft_consensus.cc:482] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:17.772681 16289 raft_consensus.cc:504] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.772832 16289 leader_election.cc:288] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627), 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117)
I20260928 12:31:17.773027 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcc5ad550e5424cb176bca9cf479466" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f741011ece4b2eba023c30586e321a" is_pre_election: true
I20260928 12:31:17.773047 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcc5ad550e5424cb176bca9cf479466" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae1f2585b8644b283c4ed0d1a569514" is_pre_election: true
I20260928 12:31:17.773118 15917 raft_consensus.cc:2454] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 0.
I20260928 12:31:17.773124 16197 raft_consensus.cc:2454] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 0.
I20260928 12:31:17.773270 15991 leader_election.cc:302] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5fedee945c49bbb2d564ade40d626b, 2ae1f2585b8644b283c4ed0d1a569514; no voters:
I20260928 12:31:17.773368 16289 raft_consensus.cc:2781] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:31:17.773396 16289 raft_consensus.cc:482] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:31:17.773418 16289 raft_consensus.cc:3037] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:17.774219 16289 raft_consensus.cc:504] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.774331 16289 leader_election.cc:288] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 election: Requested vote from peers f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627), 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117)
I20260928 12:31:17.774521 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcc5ad550e5424cb176bca9cf479466" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae1f2585b8644b283c4ed0d1a569514"
I20260928 12:31:17.774585 16197 raft_consensus.cc:3037] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:17.774600 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcc5ad550e5424cb176bca9cf479466" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f741011ece4b2eba023c30586e321a"
I20260928 12:31:17.774684 15917 raft_consensus.cc:3037] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:17.775372 16197 raft_consensus.cc:2454] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 1.
I20260928 12:31:17.775485 15917 raft_consensus.cc:2454] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 1.
I20260928 12:31:17.775642 15991 leader_election.cc:302] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5fedee945c49bbb2d564ade40d626b, f1f741011ece4b2eba023c30586e321a; no voters:
I20260928 12:31:17.775763 16289 raft_consensus.cc:2781] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:17.775808 16289 raft_consensus.cc:686] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 1 LEADER]: Becoming Leader. State: Replica: 0f5fedee945c49bbb2d564ade40d626b, State: Running, Role: LEADER
I20260928 12:31:17.775854 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.772352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:33627 (local address 127.8.13.3:47327)
0928 12:31:17.772408 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.772422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.772488 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.772658 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.772658 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.772660 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:17.772730 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.772733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.773571 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.773572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.774833 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.774837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.775586 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.775588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.775662 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.775665 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.775713 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.775762 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":2}
I20260928 12:31:17.775851 16289 consensus_queue.cc:237] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.775899 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.772509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:41365 (local address 127.8.13.2:42057)
0928 12:31:17.772568 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.772570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.772581 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:31:17.772655 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.772657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.772665 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.772714 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:17.772908 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.773574 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.775005 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.775512 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.775551 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.775563 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.775800 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.775846 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:31:17.775982 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.772431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:42057 (local address 127.8.13.3:41365)
0928 12:31:17.772539 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.772548 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.772589 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.772724 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.772725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.772728 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:17.772815 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.772816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.773637 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.773638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.774909 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.774912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.775645 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.775646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.775703 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.775705 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.775777 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.775809 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":82,"thread_start_us":57,"threads_started":1}
I20260928 12:31:17.776016 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.772428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:47327 (local address 127.8.13.1:33627)
0928 12:31:17.772469 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.772471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.772481 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:17.772576 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.772578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.772589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.772635 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:17.772804 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.773500 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.774963 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.775487 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.775555 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.775566 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.775908 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.775964 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:31:17.776249 16057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcc5ad550e5424cb176bca9cf479466" candidate_uuid: "2ae1f2585b8644b283c4ed0d1a569514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5fedee945c49bbb2d564ade40d626b" is_pre_election: true
I20260928 12:31:17.776415 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcc5ad550e5424cb176bca9cf479466" candidate_uuid: "2ae1f2585b8644b283c4ed0d1a569514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f741011ece4b2eba023c30586e321a" is_pre_election: true
I20260928 12:31:17.776501 15917 raft_consensus.cc:2379] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2ae1f2585b8644b283c4ed0d1a569514 in current term 1: Already voted for candidate 0f5fedee945c49bbb2d564ade40d626b in this term.
I20260928 12:31:17.776432 15760 catalog_manager.cc:5495] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2). New cstate: current_term: 1 leader_uuid: "0f5fedee945c49bbb2d564ade40d626b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } health_report { overall_health: UNKNOWN } } }
I20260928 12:31:17.776795 16131 leader_election.cc:302] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ae1f2585b8644b283c4ed0d1a569514; no voters: 0f5fedee945c49bbb2d564ade40d626b, f1f741011ece4b2eba023c30586e321a
I20260928 12:31:17.776914 16276 raft_consensus.cc:2726] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 28 12:31:17 dist-test-slave-g31s krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598676, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.62@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
I20260928 12:31:17.845000 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.838579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.8.13.62:33295)
0928 12:31:17.838646 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.838649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.838758 (+ 109us) server_negotiation.cc:407] Connection header received
0928 12:31:17.838817 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.838818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.838828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.838881 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:17.839297 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.839771 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.840423 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.840589 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.840642 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.842685 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:17.842686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:17.842686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:17.842694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:17.843943 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.844304 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.844305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.844305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.844354 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:17.844506 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:17.844507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:17.844508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:17.844614 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:17.844668 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:17.844806 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 12:31:17.844890 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:31:17.845067 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.838502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33295 (local address 127.0.0.1:39436)
0928 12:31:17.838682 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.838694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.838766 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.838885 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.838885 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.839177 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:17.839221 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.839223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.839836 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.839837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.840343 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.840345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.840643 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.840644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.840688 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:17.841181 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:17.841186 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:17.842594 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:17.844029 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.844038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.844042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.844239 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.844405 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:17.844405 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:17.844406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:17.844460 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:17.844665 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:17.844672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:17.844718 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:31:17.844778 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:31:17.844956 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":75,"thread_start_us":87,"threads_started":1}
I20260928 12:31:17.847098 15760 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:39436:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:31:17.847287 15760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:31:17.848683 16172 tablet_service.cc:1467] Processing CreateTablet for tablet 1a79c4eeb00746acbb5cc137cfa10fea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e4dbc4e5fa04eddaf93f535977304c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:31:17.848682 15897 tablet_service.cc:1467] Processing CreateTablet for tablet 1a79c4eeb00746acbb5cc137cfa10fea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e4dbc4e5fa04eddaf93f535977304c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:31:17.848676 16037 tablet_service.cc:1467] Processing CreateTablet for tablet 1a79c4eeb00746acbb5cc137cfa10fea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e4dbc4e5fa04eddaf93f535977304c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:31:17.848842 16037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a79c4eeb00746acbb5cc137cfa10fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:17.848842 15897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a79c4eeb00746acbb5cc137cfa10fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:17.848842 16172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a79c4eeb00746acbb5cc137cfa10fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:17.850327 16272 tablet_bootstrap.cc:492] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: Bootstrap starting.
I20260928 12:31:17.850502 16277 tablet_bootstrap.cc:492] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a: Bootstrap starting.
I20260928 12:31:17.850502 16271 tablet_bootstrap.cc:492] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b: Bootstrap starting.
I20260928 12:31:17.850687 16272 tablet_bootstrap.cc:654] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:17.850844 16271 tablet_bootstrap.cc:654] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:17.850862 16277 tablet_bootstrap.cc:654] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:17.851444 16271 tablet_bootstrap.cc:492] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b: No bootstrap required, opened a new log
I20260928 12:31:17.851478 16277 tablet_bootstrap.cc:492] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a: No bootstrap required, opened a new log
I20260928 12:31:17.851495 16271 ts_tablet_manager.cc:1397] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:17.851514 16277 ts_tablet_manager.cc:1397] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:17.851626 16271 raft_consensus.cc:348] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.851648 16277 raft_consensus.cc:348] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.851708 16271 raft_consensus.cc:374] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:17.851707 16277 raft_consensus.cc:374] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:17.851727 16277 raft_consensus.cc:729] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f741011ece4b2eba023c30586e321a, State: Initialized, Role: FOLLOWER
I20260928 12:31:17.851727 16271 raft_consensus.cc:729] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5fedee945c49bbb2d564ade40d626b, State: Initialized, Role: FOLLOWER
I20260928 12:31:17.851766 16271 consensus_queue.cc:260] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.851778 16277 consensus_queue.cc:260] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.851854 16277 ts_tablet_manager.cc:1428] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:31:17.851855 16271 ts_tablet_manager.cc:1428] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:31:17.851948 16275 tablet_replica.cc:439] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } }
I20260928 12:31:17.851986 16279 tablet_replica.cc:439] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } }
I20260928 12:31:17.852021 16275 tablet_replica.cc:442] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:31:17.852061 16279 tablet_replica.cc:442] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:31:17.852558 16272 tablet_bootstrap.cc:492] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: No bootstrap required, opened a new log
I20260928 12:31:17.852614 16272 ts_tablet_manager.cc:1397] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:31:17.852742 16272 raft_consensus.cc:348] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.852789 16272 raft_consensus.cc:374] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:17.852800 16272 raft_consensus.cc:729] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae1f2585b8644b283c4ed0d1a569514, State: Initialized, Role: FOLLOWER
I20260928 12:31:17.852836 16272 consensus_queue.cc:260] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.852919 16272 ts_tablet_manager.cc:1428] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:17.852967 16276 tablet_replica.cc:439] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } }
I20260928 12:31:17.853039 16276 tablet_replica.cc:442] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:31:17.969288 16276 raft_consensus.cc:482] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:17.969390 16276 raft_consensus.cc:504] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.969508 16276 leader_election.cc:288] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627), 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2:42057)
I20260928 12:31:17.969763 16057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a79c4eeb00746acbb5cc137cfa10fea" candidate_uuid: "2ae1f2585b8644b283c4ed0d1a569514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5fedee945c49bbb2d564ade40d626b" is_pre_election: true
I20260928 12:31:17.969765 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a79c4eeb00746acbb5cc137cfa10fea" candidate_uuid: "2ae1f2585b8644b283c4ed0d1a569514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f741011ece4b2eba023c30586e321a" is_pre_election: true
I20260928 12:31:17.969898 15917 raft_consensus.cc:2454] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae1f2585b8644b283c4ed0d1a569514 in term 0.
I20260928 12:31:17.969913 16057 raft_consensus.cc:2454] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae1f2585b8644b283c4ed0d1a569514 in term 0.
I20260928 12:31:17.970053 16131 leader_election.cc:302] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5fedee945c49bbb2d564ade40d626b, 2ae1f2585b8644b283c4ed0d1a569514; no voters:
I20260928 12:31:17.970152 16276 raft_consensus.cc:2781] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:31:17.970220 16276 raft_consensus.cc:482] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:31:17.970242 16276 raft_consensus.cc:3037] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:17.970935 16276 raft_consensus.cc:504] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.971062 16276 leader_election.cc:288] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [CANDIDATE]: Term 1 election: Requested vote from peers f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627), 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2:42057)
I20260928 12:31:17.971334 16057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a79c4eeb00746acbb5cc137cfa10fea" candidate_uuid: "2ae1f2585b8644b283c4ed0d1a569514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5fedee945c49bbb2d564ade40d626b"
I20260928 12:31:17.971412 16057 raft_consensus.cc:3037] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:17.971297 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a79c4eeb00746acbb5cc137cfa10fea" candidate_uuid: "2ae1f2585b8644b283c4ed0d1a569514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f741011ece4b2eba023c30586e321a"
I20260928 12:31:17.971508 15917 raft_consensus.cc:3037] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:17.972113 16057 raft_consensus.cc:2454] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae1f2585b8644b283c4ed0d1a569514 in term 1.
I20260928 12:31:17.972180 15917 raft_consensus.cc:2454] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae1f2585b8644b283c4ed0d1a569514 in term 1.
I20260928 12:31:17.972361 16131 leader_election.cc:302] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae1f2585b8644b283c4ed0d1a569514, f1f741011ece4b2eba023c30586e321a; no voters:
I20260928 12:31:17.972479 16276 raft_consensus.cc:2781] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:17.972611 16276 raft_consensus.cc:686] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 LEADER]: Becoming Leader. State: Replica: 2ae1f2585b8644b283c4ed0d1a569514, State: Running, Role: LEADER
I20260928 12:31:17.972679 16276 consensus_queue.cc:237] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:17.973078 16311 tablet_replica.cc:439] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: TxnStatusTablet state changed. Reason: New leader 2ae1f2585b8644b283c4ed0d1a569514. Latest consensus state: current_term: 1 leader_uuid: "2ae1f2585b8644b283c4ed0d1a569514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } }
I20260928 12:31:17.973166 16311 tablet_replica.cc:442] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:31:17.973382 16313 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:31:17.973527 16313 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:31:17.973464 15759 catalog_manager.cc:5495] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3). New cstate: current_term: 1 leader_uuid: "2ae1f2585b8644b283c4ed0d1a569514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } health_report { overall_health: HEALTHY } } }
I20260928 12:31:18.000521 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.998169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.8.13.3:42117)
0928 12:31:17.998229 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:17.998231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:17.998239 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:17.998278 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:17.998279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:17.998299 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:17.998359 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:31:17.998476 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.998971 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.999695 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.999935 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.999995 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:18.000174 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:31:18.000290 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:31:18.000365 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:18.000400 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 12:31:18.000423 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:31:18.000520 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:17.998065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:42117 (local address 127.0.0.1:50506)
0928 12:31:17.998131 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:17.998144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:17.998183 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:17.998362 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:17.998363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:17.998365 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:31:17.998419 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.998422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:17.999043 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:17.999044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:17.999619 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:17.999621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.000004 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.000005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:18.000065 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:18.000086 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:31:18.000339 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:31:18.000348 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 12:31:18.000394 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 12:31:18.000448 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:31:18.002066 15917 raft_consensus.cc:1261] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Refusing update from remote peer 2ae1f2585b8644b283c4ed0d1a569514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:18.002116 16057 raft_consensus.cc:1261] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Refusing update from remote peer 2ae1f2585b8644b283c4ed0d1a569514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:18.002283 16276 consensus_queue.cc:1035] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:18.002455 16311 consensus_queue.cc:1035] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:18.002902 16279 tablet_replica.cc:439] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a: TxnStatusTablet state changed. Reason: New leader 2ae1f2585b8644b283c4ed0d1a569514. Latest consensus state: current_term: 1 leader_uuid: "2ae1f2585b8644b283c4ed0d1a569514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } }
I20260928 12:31:18.002979 16279 tablet_replica.cc:442] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:31:18.003386 16311 tablet_replica.cc:439] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ae1f2585b8644b283c4ed0d1a569514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } }
I20260928 12:31:18.003459 16311 tablet_replica.cc:442] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:31:18.003623 16275 tablet_replica.cc:439] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b: TxnStatusTablet state changed. Reason: New leader 2ae1f2585b8644b283c4ed0d1a569514. Latest consensus state: current_term: 1 leader_uuid: "2ae1f2585b8644b283c4ed0d1a569514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } }
I20260928 12:31:18.003716 16275 tablet_replica.cc:442] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:31:18.003870 16279 tablet_replica.cc:439] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ae1f2585b8644b283c4ed0d1a569514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } }
I20260928 12:31:18.003950 16279 tablet_replica.cc:442] T 1a79c4eeb00746acbb5cc137cfa10fea P f1f741011ece4b2eba023c30586e321a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:31:18.004972 16276 tablet_replica.cc:439] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ae1f2585b8644b283c4ed0d1a569514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } }
I20260928 12:31:18.005187 16276 tablet_replica.cc:442] T 1a79c4eeb00746acbb5cc137cfa10fea P 2ae1f2585b8644b283c4ed0d1a569514: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:31:18.006212 16275 tablet_replica.cc:439] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ae1f2585b8644b283c4ed0d1a569514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } } }
I20260928 12:31:18.006312 16275 tablet_replica.cc:442] T 1a79c4eeb00746acbb5cc137cfa10fea P 0f5fedee945c49bbb2d564ade40d626b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:31:18.010195 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:18.007509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:47917 (local address 127.8.13.3:42117)
0928 12:31:18.007549 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:18.007551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:18.007624 (+ 73us) server_negotiation.cc:407] Connection header received
0928 12:31:18.007697 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:18.007698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:18.007706 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:18.007745 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:31:18.007874 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.008413 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.009552 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.009741 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.009838 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:18.009992 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:31:18.010023 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:31:18.010060 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:18.010122 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 12:31:18.010139 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:31:18.010264 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:18.007436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:42117 (local address 127.8.13.2:47917)
0928 12:31:18.007567 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:18.007578 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:18.007640 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:18.007765 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:18.007766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:18.007768 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:31:18.007822 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:18.007824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.008485 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.008485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:18.009452 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:18.009455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.009836 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.009837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:18.009896 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:18.009914 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:31:18.010071 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:31:18.010072 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:31:18.010119 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:31:18.010167 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260928 12:31:18.012423 16197 raft_consensus.cc:1261] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Refusing update from remote peer 0f5fedee945c49bbb2d564ade40d626b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:18.012640 16289 consensus_queue.cc:1035] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:18.012732 15917 raft_consensus.cc:1261] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Refusing update from remote peer 0f5fedee945c49bbb2d564ade40d626b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:18.013375 16289 consensus_queue.cc:1035] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:18.024765 16324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:18.018446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:42057 (local address 127.8.13.3:39401)
0928 12:31:18.018652 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:18.018666 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:18.018711 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:18.021972 (+ 3261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:18.021973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:18.021975 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:31:18.022052 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:18.022054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.022825 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.022827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:18.023629 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:18.023633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.024158 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.024160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:18.024252 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:18.024289 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:31:18.024539 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:31:18.024541 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:31:18.024585 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:31:18.024664 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20260928 12:31:18.024726 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:18.021697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:39401 (local address 127.8.13.2:42057)
0928 12:31:18.021788 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:18.021790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:18.021838 (+ 48us) server_negotiation.cc:407] Connection header received
0928 12:31:18.021864 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:18.021865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:18.021880 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:18.021942 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:31:18.022110 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.022751 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.023786 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.024032 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.024131 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:18.024425 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:31:18.024473 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:31:18.024535 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:18.024586 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 12:31:18.024620 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 12:31:18.034449 15760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:18.034579 15760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:18.035250 15760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0: Sending DeleteTablet for 3 replicas of tablet 8dcc5ad550e5424cb176bca9cf479466
I20260928 12:31:18.035498 15897 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dcc5ad550e5424cb176bca9cf479466 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:18 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:41146
I20260928 12:31:18.035542 16037 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dcc5ad550e5424cb176bca9cf479466 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:18 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:54516
I20260928 12:31:18.035607 16172 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dcc5ad550e5424cb176bca9cf479466 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:18 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:50494
I20260928 12:31:18.035611 16295 tablet_replica.cc:330] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a: stopping tablet replica
I20260928 12:31:18.035677 16295 raft_consensus.cc:2229] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:18.035698 16294 tablet_replica.cc:330] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514: stopping tablet replica
I20260928 12:31:18.035715 16295 raft_consensus.cc:2258] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:18.035754 16294 raft_consensus.cc:2229] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:18.035753 16327 tablet_replica.cc:330] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b: stopping tablet replica
I20260928 12:31:18.035789 16294 raft_consensus.cc:2258] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:18.035806 16327 raft_consensus.cc:2229] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:31:18.035869 16327 raft_consensus.cc:2258] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:18.036183 16327 ts_tablet_manager.cc:1905] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:18.036221 16294 ts_tablet_manager.cc:1905] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:18.037632 16295 ts_tablet_manager.cc:1905] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:18.037866 16294 ts_tablet_manager.cc:1918] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:31:18.037981 16327 ts_tablet_manager.cc:1918] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:31:18.038034 16294 log.cc:1196] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/8dcc5ad550e5424cb176bca9cf479466
I20260928 12:31:18.038064 16327 log.cc:1196] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/8dcc5ad550e5424cb176bca9cf479466
I20260928 12:31:18.038241 16294 ts_tablet_manager.cc:1939] T 8dcc5ad550e5424cb176bca9cf479466 P 2ae1f2585b8644b283c4ed0d1a569514: Deleting consensus metadata
I20260928 12:31:18.038321 16327 ts_tablet_manager.cc:1939] T 8dcc5ad550e5424cb176bca9cf479466 P 0f5fedee945c49bbb2d564ade40d626b: Deleting consensus metadata
I20260928 12:31:18.038589 15747 catalog_manager.cc:4841] TS 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117): tablet 8dcc5ad550e5424cb176bca9cf479466 (table test-table [id=c15a5f03e0b244738cc5d37a7697ba50]) successfully deleted
I20260928 12:31:18.038669 15747 catalog_manager.cc:4841] TS 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2:42057): tablet 8dcc5ad550e5424cb176bca9cf479466 (table test-table [id=c15a5f03e0b244738cc5d37a7697ba50]) successfully deleted
I20260928 12:31:18.039144 16295 ts_tablet_manager.cc:1918] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:31:18.039207 16295 log.cc:1196] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/wal/wals/8dcc5ad550e5424cb176bca9cf479466
I20260928 12:31:18.039389 16295 ts_tablet_manager.cc:1939] T 8dcc5ad550e5424cb176bca9cf479466 P f1f741011ece4b2eba023c30586e321a: Deleting consensus metadata
I20260928 12:31:18.039817 15747 catalog_manager.cc:4841] TS f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627): tablet 8dcc5ad550e5424cb176bca9cf479466 (table test-table [id=c15a5f03e0b244738cc5d37a7697ba50]) successfully deleted
Sep 28 12:31:18 dist-test-slave-g31s krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:31:18 dist-test-slave-g31s krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:18 dist-test-slave-g31s krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:31:18.060142 15828 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:18.053922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.8.13.62:33295)
0928 12:31:18.053974 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:18.053976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:18.054136 (+ 160us) server_negotiation.cc:407] Connection header received
0928 12:31:18.054215 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:18.054216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:18.054225 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:18.054299 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:18.055043 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.055548 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.056580 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.056770 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.056825 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:18.060072 (+ 3247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39448: BlockingRecv error: recv got EOF from 127.0.0.1:39448 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 12:31:18 dist-test-slave-g31s krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM
I20260928 12:31:18.070148 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:18.060952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.8.13.62:33295)
0928 12:31:18.061010 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:18.061012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:18.061022 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:18.061069 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:18.061070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:18.061081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:18.061148 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:18.061564 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.062336 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.063331 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.063687 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.063794 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:18.066658 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:18.066661 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:18.066661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:18.066672 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:18.068781 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:18.069199 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:18.069200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:18.069201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:18.069307 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:18.069537 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:18.069538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:18.069539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:18.069690 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:18.069783 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:18.069950 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 12:31:18.070050 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:31:18.071771 15760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:31:18.071967 15760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:31:18.073307 16037 tablet_service.cc:1467] Processing CreateTablet for tablet 4604fd0c07fa4723bd350b53e60e8c7d (DEFAULT_TABLE table=test-table [id=0964472c13604d8d9f0f1c342195739b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:18.073362 16172 tablet_service.cc:1467] Processing CreateTablet for tablet 4604fd0c07fa4723bd350b53e60e8c7d (DEFAULT_TABLE table=test-table [id=0964472c13604d8d9f0f1c342195739b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:18.073372 15897 tablet_service.cc:1467] Processing CreateTablet for tablet 4604fd0c07fa4723bd350b53e60e8c7d (DEFAULT_TABLE table=test-table [id=0964472c13604d8d9f0f1c342195739b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:18.073472 16172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4604fd0c07fa4723bd350b53e60e8c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:18.073472 16037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4604fd0c07fa4723bd350b53e60e8c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:18.073498 15897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4604fd0c07fa4723bd350b53e60e8c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:18.074950 16271 tablet_bootstrap.cc:492] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b: Bootstrap starting.
I20260928 12:31:18.074957 16277 tablet_bootstrap.cc:492] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a: Bootstrap starting.
I20260928 12:31:18.075284 16277 tablet_bootstrap.cc:654] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:18.075284 16271 tablet_bootstrap.cc:654] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:18.075587 16272 tablet_bootstrap.cc:492] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514: Bootstrap starting.
I20260928 12:31:18.075884 16271 tablet_bootstrap.cc:492] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b: No bootstrap required, opened a new log
I20260928 12:31:18.075937 16271 ts_tablet_manager.cc:1397] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:18.075965 16272 tablet_bootstrap.cc:654] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:18.076062 16271 raft_consensus.cc:348] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.076143 16271 raft_consensus.cc:374] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:18.076161 16271 raft_consensus.cc:729] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5fedee945c49bbb2d564ade40d626b, State: Initialized, Role: FOLLOWER
I20260928 12:31:18.076210 16271 consensus_queue.cc:260] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.076298 16271 ts_tablet_manager.cc:1428] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:18.076597 16277 tablet_bootstrap.cc:492] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a: No bootstrap required, opened a new log
I20260928 12:31:18.076642 16277 ts_tablet_manager.cc:1397] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:31:18.076768 16277 raft_consensus.cc:348] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.076836 16277 raft_consensus.cc:374] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:18.076874 16277 raft_consensus.cc:729] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f741011ece4b2eba023c30586e321a, State: Initialized, Role: FOLLOWER
I20260928 12:31:18.076913 16277 consensus_queue.cc:260] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.076951 16272 tablet_bootstrap.cc:492] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514: No bootstrap required, opened a new log
I20260928 12:31:18.076985 16272 ts_tablet_manager.cc:1397] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:18.076997 16277 ts_tablet_manager.cc:1428] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:18.077142 16272 raft_consensus.cc:348] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.077207 16272 raft_consensus.cc:374] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:18.077227 16272 raft_consensus.cc:729] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae1f2585b8644b283c4ed0d1a569514, State: Initialized, Role: FOLLOWER
I20260928 12:31:18.077260 16272 consensus_queue.cc:260] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.077332 16272 ts_tablet_manager.cc:1428] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:18.278936 16275 raft_consensus.cc:482] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:18.279052 16275 raft_consensus.cc:504] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.279215 16275 leader_election.cc:288] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627), 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117)
I20260928 12:31:18.279490 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4604fd0c07fa4723bd350b53e60e8c7d" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae1f2585b8644b283c4ed0d1a569514" is_pre_election: true
I20260928 12:31:18.279490 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4604fd0c07fa4723bd350b53e60e8c7d" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f741011ece4b2eba023c30586e321a" is_pre_election: true
I20260928 12:31:18.279603 16197 raft_consensus.cc:2454] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 0.
I20260928 12:31:18.279603 15917 raft_consensus.cc:2454] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 0.
I20260928 12:31:18.279757 15991 leader_election.cc:302] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5fedee945c49bbb2d564ade40d626b, 2ae1f2585b8644b283c4ed0d1a569514; no voters:
I20260928 12:31:18.279861 16275 raft_consensus.cc:2781] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:31:18.279909 16275 raft_consensus.cc:482] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:31:18.279929 16275 raft_consensus.cc:3037] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:18.280627 16275 raft_consensus.cc:504] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.280748 16275 leader_election.cc:288] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 election: Requested vote from peers f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627), 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117)
I20260928 12:31:18.280949 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4604fd0c07fa4723bd350b53e60e8c7d" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f741011ece4b2eba023c30586e321a"
I20260928 12:31:18.280964 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4604fd0c07fa4723bd350b53e60e8c7d" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae1f2585b8644b283c4ed0d1a569514"
I20260928 12:31:18.281040 16197 raft_consensus.cc:3037] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:18.281040 15917 raft_consensus.cc:3037] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:18.281774 16197 raft_consensus.cc:2454] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 1.
I20260928 12:31:18.281782 15917 raft_consensus.cc:2454] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 1.
I20260928 12:31:18.281980 15991 leader_election.cc:302] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5fedee945c49bbb2d564ade40d626b, f1f741011ece4b2eba023c30586e321a; no voters:
I20260928 12:31:18.282089 16275 raft_consensus.cc:2781] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:18.282156 16275 raft_consensus.cc:686] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 1 LEADER]: Becoming Leader. State: Replica: 0f5fedee945c49bbb2d564ade40d626b, State: Running, Role: LEADER
I20260928 12:31:18.282253 16275 consensus_queue.cc:237] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.282845 15760 catalog_manager.cc:5495] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2). New cstate: current_term: 1 leader_uuid: "0f5fedee945c49bbb2d564ade40d626b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } health_report { overall_health: UNKNOWN } } }
I20260928 12:31:18.307175 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:18.305199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.8.13.2:42057)
0928 12:31:18.305258 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:18.305260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:18.305268 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:18.305291 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:18.305292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:18.305304 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:18.305349 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:31:18.305475 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.305976 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.306667 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:18.306852 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:18.306902 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:18.306991 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:31:18.307018 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:31:18.307046 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:18.307082 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 12:31:18.307096 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:31:18.307955 16197 raft_consensus.cc:1261] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Refusing update from remote peer 0f5fedee945c49bbb2d564ade40d626b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:18.307960 15917 raft_consensus.cc:1261] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Refusing update from remote peer 0f5fedee945c49bbb2d564ade40d626b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:18.308197 16281 consensus_queue.cc:1035] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:18.308310 16275 consensus_queue.cc:1035] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:18.309247 16284 mvcc.cc:205] Tried to move back new op lower bound from 7334292186347802624 to 7334292186244603904. Current Snapshot: MvccSnapshot[applied={T|T < 7334292186347802624}]
I20260928 12:31:18.310503 15759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:18.310590 15759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:18.311110 15759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0: Sending DeleteTablet for 3 replicas of tablet 4604fd0c07fa4723bd350b53e60e8c7d
I20260928 12:31:18.311312 15897 tablet_service.cc:1514] Processing DeleteTablet for tablet 4604fd0c07fa4723bd350b53e60e8c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:18 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:41146
I20260928 12:31:18.311368 16037 tablet_service.cc:1514] Processing DeleteTablet for tablet 4604fd0c07fa4723bd350b53e60e8c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:18 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:54516
I20260928 12:31:18.311406 16295 tablet_replica.cc:330] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a: stopping tablet replica
I20260928 12:31:18.311447 16295 raft_consensus.cc:2229] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:18.311460 16172 tablet_service.cc:1514] Processing DeleteTablet for tablet 4604fd0c07fa4723bd350b53e60e8c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:18 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:50494
I20260928 12:31:18.311488 16295 raft_consensus.cc:2258] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:18.311545 16327 tablet_replica.cc:330] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b: stopping tablet replica
I20260928 12:31:18.311599 16327 raft_consensus.cc:2229] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:31:18.311542 15759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 12:31:18.311655 16327 raft_consensus.cc:2258] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 12:31:18.311668 15759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:31:18.311718 16294 tablet_replica.cc:330] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514: stopping tablet replica
I20260928 12:31:18.311761 16294 raft_consensus.cc:2229] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:18.311815 16294 raft_consensus.cc:2258] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:18.312175 16294 ts_tablet_manager.cc:1905] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:18.312206 16327 ts_tablet_manager.cc:1905] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:18.313659 16295 ts_tablet_manager.cc:1905] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:18.313894 16327 ts_tablet_manager.cc:1918] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:18.313967 16327 log.cc:1196] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/4604fd0c07fa4723bd350b53e60e8c7d
I20260928 12:31:18.313988 16037 tablet_service.cc:1467] Processing CreateTablet for tablet 6628f0584a9b46aeaa3d296451ac6427 (DEFAULT_TABLE table=test-table [id=7f597462951942f0bfd2ac1c8e56767b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:18.314042 16172 tablet_service.cc:1467] Processing CreateTablet for tablet 6628f0584a9b46aeaa3d296451ac6427 (DEFAULT_TABLE table=test-table [id=7f597462951942f0bfd2ac1c8e56767b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:18.314093 16037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6628f0584a9b46aeaa3d296451ac6427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:18.314118 16172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6628f0584a9b46aeaa3d296451ac6427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:18.314143 16327 ts_tablet_manager.cc:1939] T 4604fd0c07fa4723bd350b53e60e8c7d P 0f5fedee945c49bbb2d564ade40d626b: Deleting consensus metadata
I20260928 12:31:18.314383 15897 tablet_service.cc:1467] Processing CreateTablet for tablet 6628f0584a9b46aeaa3d296451ac6427 (DEFAULT_TABLE table=test-table [id=7f597462951942f0bfd2ac1c8e56767b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:18.314486 15897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6628f0584a9b46aeaa3d296451ac6427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:18.315061 16295 ts_tablet_manager.cc:1918] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:18.315107 16295 log.cc:1196] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-0/wal/wals/4604fd0c07fa4723bd350b53e60e8c7d
I20260928 12:31:18.315187 16272 tablet_bootstrap.cc:492] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514: Bootstrap starting.
I20260928 12:31:18.315284 16295 ts_tablet_manager.cc:1939] T 4604fd0c07fa4723bd350b53e60e8c7d P f1f741011ece4b2eba023c30586e321a: Deleting consensus metadata
I20260928 12:31:18.315498 16272 tablet_bootstrap.cc:654] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:18.315623 16277 tablet_bootstrap.cc:492] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a: Bootstrap starting.
I20260928 12:31:18.315724 15747 catalog_manager.cc:4841] TS f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627): tablet 4604fd0c07fa4723bd350b53e60e8c7d (table test-table [id=0964472c13604d8d9f0f1c342195739b]) successfully deleted
I20260928 12:31:18.315817 15747 catalog_manager.cc:4841] TS 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2:42057): tablet 4604fd0c07fa4723bd350b53e60e8c7d (table test-table [id=0964472c13604d8d9f0f1c342195739b]) successfully deleted
I20260928 12:31:18.315833 16271 tablet_bootstrap.cc:492] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b: Bootstrap starting.
I20260928 12:31:18.315912 16294 ts_tablet_manager.cc:1918] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:31:18.315964 16294 log.cc:1196] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/4604fd0c07fa4723bd350b53e60e8c7d
I20260928 12:31:18.316110 16271 tablet_bootstrap.cc:654] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:18.316135 16294 ts_tablet_manager.cc:1939] T 4604fd0c07fa4723bd350b53e60e8c7d P 2ae1f2585b8644b283c4ed0d1a569514: Deleting consensus metadata
I20260928 12:31:18.316169 16277 tablet_bootstrap.cc:654] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:18.316366 15747 catalog_manager.cc:4841] TS 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117): tablet 4604fd0c07fa4723bd350b53e60e8c7d (table test-table [id=0964472c13604d8d9f0f1c342195739b]) successfully deleted
I20260928 12:31:18.316444 16272 tablet_bootstrap.cc:492] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514: No bootstrap required, opened a new log
I20260928 12:31:18.316489 16272 ts_tablet_manager.cc:1397] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:18.316597 16272 raft_consensus.cc:348] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.316648 16271 tablet_bootstrap.cc:492] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b: No bootstrap required, opened a new log
I20260928 12:31:18.316655 16272 raft_consensus.cc:374] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:18.316673 16272 raft_consensus.cc:729] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae1f2585b8644b283c4ed0d1a569514, State: Initialized, Role: FOLLOWER
I20260928 12:31:18.316679 16271 ts_tablet_manager.cc:1397] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:31:18.316721 16272 consensus_queue.cc:260] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.316766 16271 raft_consensus.cc:348] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.316811 16272 ts_tablet_manager.cc:1428] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:18.316835 16271 raft_consensus.cc:374] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:18.316854 16271 raft_consensus.cc:729] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5fedee945c49bbb2d564ade40d626b, State: Initialized, Role: FOLLOWER
I20260928 12:31:18.316905 16271 consensus_queue.cc:260] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.316988 16271 ts_tablet_manager.cc:1428] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:18.317553 16277 tablet_bootstrap.cc:492] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a: No bootstrap required, opened a new log
I20260928 12:31:18.317595 16277 ts_tablet_manager.cc:1397] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:31:18.317744 16277 raft_consensus.cc:348] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.317829 16277 raft_consensus.cc:374] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:18.317859 16277 raft_consensus.cc:729] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f741011ece4b2eba023c30586e321a, State: Initialized, Role: FOLLOWER
I20260928 12:31:18.317899 16277 consensus_queue.cc:260] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.318003 16277 ts_tablet_manager.cc:1428] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:18.361456 16275 raft_consensus.cc:482] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:18.361562 16275 raft_consensus.cc:504] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.361714 16275 leader_election.cc:288] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627), 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117)
I20260928 12:31:18.362036 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6628f0584a9b46aeaa3d296451ac6427" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f741011ece4b2eba023c30586e321a" is_pre_election: true
I20260928 12:31:18.362043 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6628f0584a9b46aeaa3d296451ac6427" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae1f2585b8644b283c4ed0d1a569514" is_pre_election: true
I20260928 12:31:18.362140 16197 raft_consensus.cc:2454] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 0.
I20260928 12:31:18.362140 15917 raft_consensus.cc:2454] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 0.
I20260928 12:31:18.362300 15991 leader_election.cc:302] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5fedee945c49bbb2d564ade40d626b, 2ae1f2585b8644b283c4ed0d1a569514; no voters:
I20260928 12:31:18.362448 16275 raft_consensus.cc:2781] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:31:18.362496 16275 raft_consensus.cc:482] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:31:18.362521 16275 raft_consensus.cc:3037] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:18.363385 16275 raft_consensus.cc:504] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.363517 16275 leader_election.cc:288] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 election: Requested vote from peers f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627), 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117)
I20260928 12:31:18.363734 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6628f0584a9b46aeaa3d296451ac6427" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae1f2585b8644b283c4ed0d1a569514"
I20260928 12:31:18.363811 16197 raft_consensus.cc:3037] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:18.363710 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6628f0584a9b46aeaa3d296451ac6427" candidate_uuid: "0f5fedee945c49bbb2d564ade40d626b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f741011ece4b2eba023c30586e321a"
I20260928 12:31:18.363938 15917 raft_consensus.cc:3037] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:18.364593 15917 raft_consensus.cc:2454] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 1.
I20260928 12:31:18.364627 16197 raft_consensus.cc:2454] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f5fedee945c49bbb2d564ade40d626b in term 1.
I20260928 12:31:18.364789 15991 leader_election.cc:302] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5fedee945c49bbb2d564ade40d626b, 2ae1f2585b8644b283c4ed0d1a569514; no voters:
I20260928 12:31:18.364890 16275 raft_consensus.cc:2781] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:18.364944 16275 raft_consensus.cc:686] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 1 LEADER]: Becoming Leader. State: Replica: 0f5fedee945c49bbb2d564ade40d626b, State: Running, Role: LEADER
I20260928 12:31:18.364995 16275 consensus_queue.cc:237] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } }
I20260928 12:31:18.365595 15759 catalog_manager.cc:5495] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2). New cstate: current_term: 1 leader_uuid: "0f5fedee945c49bbb2d564ade40d626b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f5fedee945c49bbb2d564ade40d626b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 } health_report { overall_health: UNKNOWN } } }
I20260928 12:31:18.380991 15917 raft_consensus.cc:1261] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Refusing update from remote peer 0f5fedee945c49bbb2d564ade40d626b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:18.381011 16197 raft_consensus.cc:1261] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Refusing update from remote peer 0f5fedee945c49bbb2d564ade40d626b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:31:18.381225 16289 consensus_queue.cc:1035] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae1f2585b8644b283c4ed0d1a569514" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:18.381353 16275 consensus_queue.cc:1035] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f741011ece4b2eba023c30586e321a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:18.392056 15759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:18.392166 15759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:18.392899 15759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa4838cd6de34dc5b83265ac40539ae0: Sending DeleteTablet for 3 replicas of tablet 6628f0584a9b46aeaa3d296451ac6427
I20260928 12:31:18.393162 15897 tablet_service.cc:1514] Processing DeleteTablet for tablet 6628f0584a9b46aeaa3d296451ac6427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:18 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:41146
I20260928 12:31:18.393195 16037 tablet_service.cc:1514] Processing DeleteTablet for tablet 6628f0584a9b46aeaa3d296451ac6427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:18 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:54516
I20260928 12:31:18.393296 16327 tablet_replica.cc:330] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b: stopping tablet replica
I20260928 12:31:18.393296 16295 tablet_replica.cc:330] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a: stopping tablet replica
I20260928 12:31:18.393306 16172 tablet_service.cc:1514] Processing DeleteTablet for tablet 6628f0584a9b46aeaa3d296451ac6427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:18 UTC) from {username='oryx', principal='oryx/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:50494
I20260928 12:31:18.393354 16327 raft_consensus.cc:2229] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:31:18.393360 16295 raft_consensus.cc:2229] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:18.393399 16295 raft_consensus.cc:2258] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:18.393410 16294 tablet_replica.cc:330] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514: stopping tablet replica
I20260928 12:31:18.393426 16327 raft_consensus.cc:2258] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:18.393464 16294 raft_consensus.cc:2229] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:18.393499 16294 raft_consensus.cc:2258] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:18.393702 16295 ts_tablet_manager.cc:1905] T 6628f0584a9b46aeaa3d296451ac6427 P f1f741011ece4b2eba023c30586e321a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:18.393791 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 15832
I20260928 12:31:18.393934 16327 ts_tablet_manager.cc:1905] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:18.395581 16327 ts_tablet_manager.cc:1918] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:31:18.395650 16327 log.cc:1196] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/6628f0584a9b46aeaa3d296451ac6427
I20260928 12:31:18.395825 16327 ts_tablet_manager.cc:1939] T 6628f0584a9b46aeaa3d296451ac6427 P 0f5fedee945c49bbb2d564ade40d626b: Deleting consensus metadata
I20260928 12:31:18.396175 15747 catalog_manager.cc:4841] TS 0f5fedee945c49bbb2d564ade40d626b (127.8.13.2:42057): tablet 6628f0584a9b46aeaa3d296451ac6427 (table test-table [id=7f597462951942f0bfd2ac1c8e56767b]) successfully deleted
I20260928 12:31:18.396479 16294 ts_tablet_manager.cc:1905] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:18.397673 16294 ts_tablet_manager.cc:1918] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:31:18.397744 16294 log.cc:1196] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/6628f0584a9b46aeaa3d296451ac6427
I20260928 12:31:18.397965 16294 ts_tablet_manager.cc:1939] T 6628f0584a9b46aeaa3d296451ac6427 P 2ae1f2585b8644b283c4ed0d1a569514: Deleting consensus metadata
I20260928 12:31:18.398293 15747 catalog_manager.cc:4841] TS 2ae1f2585b8644b283c4ed0d1a569514 (127.8.13.3:42117): tablet 6628f0584a9b46aeaa3d296451ac6427 (table test-table [id=7f597462951942f0bfd2ac1c8e56767b]) successfully deleted
W20260928 12:31:18.400965 15747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:33627 (error 108)
I20260928 12:31:18.401142 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 15972
I20260928 12:31:18.406483 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:18.406137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:33627 (local address 127.0.0.1:41158)
0928 12:31:18.406329 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:18.406424 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:33627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
W20260928 12:31:18.406708 15747 catalog_manager.cc:4584] TS f1f741011ece4b2eba023c30586e321a (127.8.13.1:33627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6628f0584a9b46aeaa3d296451ac6427: Network error: Client connection negotiation failed: client connection to 127.8.13.1:33627: connect: Connection refused (error 111)
I20260928 12:31:18.407828 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 16112
I20260928 12:31:18.413029 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 15728
2026-09-28T12:31:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2797 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:31:18 dist-test-slave-g31s krb5kdc[16357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:31:18 dist-test-slave-g31s krb5kdc[16357](info): set up 2 sockets
Sep 28 12:31:18 dist-test-slave-g31s krb5kdc[16357](info): commencing operation
krb5kdc: starting...
W20260928 12:31:19.366523 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:31:19 dist-test-slave-g31s krb5kdc[16357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:31:19.487294 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45365
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:35455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:45365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:19.570040 16370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:19.570302 16370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:19.570329 16370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:19.571543 16370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:19.571588 16370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:19.571602 16370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:19.571614 16370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:19.571624 16370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:19.573200 16370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:45365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:19.573437 16370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:19.573628 16370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:19.578482 16376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:19.578562 16370 server_base.cc:1034] running on GCE node
W20260928 12:31:19.578482 16378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:19.578482 16375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:19.578925 16370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:19.579231 16370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:19.580410 16370 hybrid_clock.cc:648] HybridClock initialized: now 1790598679580396 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T12:31:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1171 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:31:19 dist-test-slave-g31s krb5kdc[16383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:31:19 dist-test-slave-g31s krb5kdc[16383](info): set up 2 sockets
Sep 28 12:31:19 dist-test-slave-g31s krb5kdc[16383](info): commencing operation
krb5kdc: starting...
W20260928 12:31:20.538167 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:31:20 dist-test-slave-g31s krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:31:20.657233 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:40511
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:40511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:20.741183 16396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:20.741449 16396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:20.741484 16396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:20.742856 16396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:20.742923 16396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:20.742942 16396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:20.742960 16396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:20.742975 16396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:20.745088 16396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:40511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:20.745572 16396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:20.745939 16396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:20.752234 16401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:20.752234 16402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:20.752234 16404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:20.753091 16396 server_base.cc:1034] running on GCE node
I20260928 12:31:20.753475 16396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:20.753818 16396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:20.755018 16396 hybrid_clock.cc:648] HybridClock initialized: now 1790598680754979 us; error 58 us; skew 500 ppm
Sep 28 12:31:20 dist-test-slave-g31s krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598680, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:20.764060 16396 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:31:20.764572 16396 webserver.cc:460] Webserver started at http://127.8.13.62:38947/ using document root <none> and password file <none>
I20260928 12:31:20.764820 16396 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:20.764878 16396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:20.765009 16396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:20.766196 16396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "b2aee5880a264b1cbf83523106b242a3"
format_stamp: "Formatted at 2026-09-28 12:31:20 on dist-test-slave-g31s"
server_key: "7a637791d9eff5a7f5b9e7b12a72227d"
server_key_iv: "bc0fe07c1599275467f38f68f57ccf66"
server_key_version: "encryptionkey@0"
I20260928 12:31:20.766544 16396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "b2aee5880a264b1cbf83523106b242a3"
format_stamp: "Formatted at 2026-09-28 12:31:20 on dist-test-slave-g31s"
server_key: "7a637791d9eff5a7f5b9e7b12a72227d"
server_key_iv: "bc0fe07c1599275467f38f68f57ccf66"
server_key_version: "encryptionkey@0"
I20260928 12:31:20.767894 16396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:20.768914 16411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:20.769187 16396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:20.769294 16396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "b2aee5880a264b1cbf83523106b242a3"
format_stamp: "Formatted at 2026-09-28 12:31:20 on dist-test-slave-g31s"
server_key: "7a637791d9eff5a7f5b9e7b12a72227d"
server_key_iv: "bc0fe07c1599275467f38f68f57ccf66"
server_key_version: "encryptionkey@0"
I20260928 12:31:20.769361 16396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:20.778398 16396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:20.781183 16396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:20.781312 16396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:20.785112 16396 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:40511
I20260928 12:31:20.785153 16463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:40511 every 8 connection(s)
I20260928 12:31:20.785480 16396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:20.786379 16464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:20.788792 16464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3: Bootstrap starting.
I20260928 12:31:20.789445 16464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:20.789727 16464 log.cc:824] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:20.790809 16464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3: No bootstrap required, opened a new log
I20260928 12:31:20.792687 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 16396
I20260928 12:31:20.792776 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:20.792726 16464 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } }
I20260928 12:31:20.792945 8244 external_mini_cluster.cc:1428] Setting key 50495dbbf3c5df8ddf93cd9b00580857
I20260928 12:31:20.792933 16464 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:20.792975 16464 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2aee5880a264b1cbf83523106b242a3, State: Initialized, Role: FOLLOWER
I20260928 12:31:20.793135 16464 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } }
I20260928 12:31:20.793223 16464 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:20.793260 16464 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:20.793303 16464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:20.794106 16464 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } }
I20260928 12:31:20.794221 16464 leader_election.cc:302] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2aee5880a264b1cbf83523106b242a3; no voters:
I20260928 12:31:20.794396 16464 leader_election.cc:288] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:20.794437 16469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:20.794615 16469 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 1 LEADER]: Becoming Leader. State: Replica: b2aee5880a264b1cbf83523106b242a3, State: Running, Role: LEADER
I20260928 12:31:20.794655 16464 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:20.794736 16469 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } }
I20260928 12:31:20.795202 16470 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2aee5880a264b1cbf83523106b242a3. Latest consensus state: current_term: 1 leader_uuid: "b2aee5880a264b1cbf83523106b242a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } } }
I20260928 12:31:20.795382 16470 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:20.796036 16478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:20.796116 16471 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2aee5880a264b1cbf83523106b242a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } } }
I20260928 12:31:20.796202 16471 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:20.796695 16478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:20.798393 16478 catalog_manager.cc:1347] Generated new cluster ID: 873ebe4849414f65b51e0d7a4ceb7b45
I20260928 12:31:20.798440 16478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:31:20 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:20.804919 16478 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:20.805651 16478 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:20.805871 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:20.793408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.8.13.62:40511)
0928 12:31:20.793624 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:20.793635 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:20.793669 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:31:20.794081 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:20.794089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:20.794143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:20.794276 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:20.795137 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:20.796582 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:20.797384 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:20.797870 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:20.797995 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:20.800867 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:20.800872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:20.800872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:20.800887 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:20.803984 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:20.804468 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:20.804472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:20.804472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:20.804597 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:20.804847 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:20.804848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:20.804849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:20.805151 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:20.805302 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:20.805425 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 12:31:20.805610 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
I20260928 12:31:20.811376 16478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3: Generated new TSK 0
I20260928 12:31:20.811578 16478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:31:20.842900 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40511
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:20.926074 16490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:20.926309 16490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:20.926342 16490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:20.927573 16490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:20.927616 16490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:20.927652 16490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:20.929162 16490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:20.929404 16490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:20.929612 16490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:20.934549 16498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:20.934585 16495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:20.934831 16496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:20.935458 16490 server_base.cc:1034] running on GCE node
I20260928 12:31:20.935699 16490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:20.935969 16490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:20.937153 16490 hybrid_clock.cc:648] HybridClock initialized: now 1790598680937137 us; error 34 us; skew 500 ppm
Sep 28 12:31:20 dist-test-slave-g31s krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598680, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:20.940482 16490 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:31:20.941026 16490 webserver.cc:460] Webserver started at http://127.8.13.1:42199/ using document root <none> and password file <none>
I20260928 12:31:20.941269 16490 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:20.941323 16490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:20.941460 16490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:20.942623 16490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "8c10ad764f4e41aca1397c8b88617d5b"
format_stamp: "Formatted at 2026-09-28 12:31:20 on dist-test-slave-g31s"
server_key: "b36f4d8d9d69b3c437ba1c14e48b8f9b"
server_key_iv: "7b415b3d7cbc09548547466d136cd81d"
server_key_version: "encryptionkey@0"
I20260928 12:31:20.942962 16490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "8c10ad764f4e41aca1397c8b88617d5b"
format_stamp: "Formatted at 2026-09-28 12:31:20 on dist-test-slave-g31s"
server_key: "b36f4d8d9d69b3c437ba1c14e48b8f9b"
server_key_iv: "7b415b3d7cbc09548547466d136cd81d"
server_key_version: "encryptionkey@0"
I20260928 12:31:20.944300 16490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:31:20.945173 16505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:20.945384 16490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:20.945475 16490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "8c10ad764f4e41aca1397c8b88617d5b"
format_stamp: "Formatted at 2026-09-28 12:31:20 on dist-test-slave-g31s"
server_key: "b36f4d8d9d69b3c437ba1c14e48b8f9b"
server_key_iv: "7b415b3d7cbc09548547466d136cd81d"
server_key_version: "encryptionkey@0"
I20260928 12:31:20.945544 16490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:20.967617 16490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:20.969854 16490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:20.970005 16490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:20.970263 16490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:20.970578 16490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:20.970613 16490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:20.970643 16490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:20.970664 16490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:20.976979 16490 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:36017
I20260928 12:31:20.977041 16618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:36017 every 8 connection(s)
I20260928 12:31:20.977363 16490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:31:20.978403 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 16490
I20260928 12:31:20.978510 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:31:20.978686 8244 external_mini_cluster.cc:1428] Setting key 994567a7b74399ee1d90363ecea1a5b1
Sep 28 12:31:20 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598680, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
I20260928 12:31:20.987768 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:20.978265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:53729 (local address 127.8.13.62:40511)
0928 12:31:20.978861 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:20.978863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:20.978873 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:20.979084 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:20.979086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:20.979097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:20.979153 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:20.980159 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:20.980670 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:20.981706 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:20.981975 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:20.982036 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:20.984703 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:20.984705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:20.984705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:20.984720 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:20.986299 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:20.986785 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:20.986786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:20.986786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:20.986840 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:20.987069 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:20.987070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:20.987070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:20.987189 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:20.987247 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:20.987535 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 12:31:20.987672 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570}
I20260928 12:31:20.987924 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:20.978174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:53729)
0928 12:31:20.978348 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:20.978389 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:20.978979 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:20.979230 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:20.979234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:20.979853 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:20.980057 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:20.980062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:20.980771 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:20.980772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:20.981608 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:20.981610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:20.982064 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:20.982065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:20.982202 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:20.982877 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:20.982886 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:20.984587 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:20.986406 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:20.986418 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:20.986421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:20.986710 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:20.986897 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:20.986898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:20.986898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:20.987012 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:20.987246 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:20.987253 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:20.987409 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 12:31:20.987504 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 12:31:20.987686 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260928 12:31:20.988384 16619 heartbeater.cc:347] Connected to a master server at 127.8.13.62:40511
I20260928 12:31:20.988492 16619 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:20.988674 16619 heartbeater.cc:511] Master 127.8.13.62:40511 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
I20260928 12:31:20.989182 16428 ts_manager.cc:194] Registered new tserver with Master: 8c10ad764f4e41aca1397c8b88617d5b (127.8.13.1:36017)
I20260928 12:31:20.989867 16428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:53729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:31:21.005257 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40511
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:21.089716 16624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:21.089986 16624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:21.090013 16624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:21.091266 16624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:21.091317 16624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:21.091347 16624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:21.092891 16624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:21.093112 16624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:21.093310 16624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:21.098467 16629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:21.098537 16624 server_base.cc:1034] running on GCE node
W20260928 12:31:21.098469 16632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:21.098469 16630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:21.098917 16624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:21.099185 16624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:21.100343 16624 hybrid_clock.cc:648] HybridClock initialized: now 1790598681100330 us; error 30 us; skew 500 ppm
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598681, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:21.103070 16624 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:31:21.103526 16624 webserver.cc:460] Webserver started at http://127.8.13.2:33165/ using document root <none> and password file <none>
I20260928 12:31:21.103751 16624 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:21.103801 16624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:21.103937 16624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:21.105046 16624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "2cba36094dc1488eb596c34ad0bff674"
format_stamp: "Formatted at 2026-09-28 12:31:21 on dist-test-slave-g31s"
server_key: "cd8c369f98a9fc4237acd11137fdd7c5"
server_key_iv: "dc10599bc77d22aacd270d88fe9d13a3"
server_key_version: "encryptionkey@0"
I20260928 12:31:21.105388 16624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "2cba36094dc1488eb596c34ad0bff674"
format_stamp: "Formatted at 2026-09-28 12:31:21 on dist-test-slave-g31s"
server_key: "cd8c369f98a9fc4237acd11137fdd7c5"
server_key_iv: "dc10599bc77d22aacd270d88fe9d13a3"
server_key_version: "encryptionkey@0"
I20260928 12:31:21.106761 16624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:31:21.107578 16639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.107750 16624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:21.107832 16624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "2cba36094dc1488eb596c34ad0bff674"
format_stamp: "Formatted at 2026-09-28 12:31:21 on dist-test-slave-g31s"
server_key: "cd8c369f98a9fc4237acd11137fdd7c5"
server_key_iv: "dc10599bc77d22aacd270d88fe9d13a3"
server_key_version: "encryptionkey@0"
I20260928 12:31:21.107898 16624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:21.160557 16624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:21.163746 16624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:21.163955 16624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:21.164219 16624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:21.164568 16624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:21.164608 16624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.164642 16624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:21.164659 16624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.171327 16624 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:43531
I20260928 12:31:21.171375 16752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:43531 every 8 connection(s)
I20260928 12:31:21.171702 16624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598681, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:21.181420 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 16624
I20260928 12:31:21.181541 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:31:21.181730 8244 external_mini_cluster.cc:1428] Setting key e7a61cb5b283d6681d86fb3b1dd7fdef
I20260928 12:31:21.182271 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:21.172439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:50727 (local address 127.8.13.62:40511)
0928 12:31:21.172515 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:21.172518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:21.172704 (+ 186us) server_negotiation.cc:407] Connection header received
0928 12:31:21.173191 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:21.173192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:21.173203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:21.173255 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:21.174269 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.174773 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.175846 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.176056 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.176119 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.179093 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:21.179095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:21.179096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:21.179104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:21.180507 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:21.180974 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:21.180975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:21.180975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:21.181037 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:21.181359 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:21.181361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:21.181361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:21.181518 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:21.181562 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:21.182066 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 12:31:21.182167 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:31:21.182469 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:21.172416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:50727)
0928 12:31:21.172580 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:21.172618 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:21.173108 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:21.173319 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:21.173323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:21.173944 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:21.174157 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.174162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.174886 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.174887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.175736 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.175739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.176156 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.176157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.176287 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.176941 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:21.176954 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:21.178985 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:21.180621 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:21.180630 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:21.180632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:21.180900 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:21.181111 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:21.181112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:21.181112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:21.181268 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:21.181668 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:21.181674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:21.181896 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 12:31:21.182024 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 12:31:21.182223 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260928 12:31:21.183712 16753 heartbeater.cc:347] Connected to a master server at 127.8.13.62:40511
I20260928 12:31:21.183861 16753 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:21.184098 16753 heartbeater.cc:511] Master 127.8.13.62:40511 requested a full tablet report, sending...
I20260928 12:31:21.184509 16428 ts_manager.cc:194] Registered new tserver with Master: 2cba36094dc1488eb596c34ad0bff674 (127.8.13.2:43531)
I20260928 12:31:21.184890 16428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:50727
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:31:21.209084 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40511
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:21.294543 16758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:21.294827 16758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:21.294867 16758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:21.297497 16758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:21.297571 16758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:21.297607 16758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:21.299937 16758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:21.300199 16758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:21.300441 16758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:21.305624 16766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:21.305601 16758 server_base.cc:1034] running on GCE node
W20260928 12:31:21.305595 16763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:21.305605 16764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:21.306095 16758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:21.306380 16758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:21.307543 16758 hybrid_clock.cc:648] HybridClock initialized: now 1790598681307524 us; error 34 us; skew 500 ppm
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598681, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:21.310040 16758 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:31:21.310534 16758 webserver.cc:460] Webserver started at http://127.8.13.3:37037/ using document root <none> and password file <none>
I20260928 12:31:21.310786 16758 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:21.310838 16758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:21.310968 16758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:21.312088 16758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "d693500a9ae54bc988c9753c795142f6"
format_stamp: "Formatted at 2026-09-28 12:31:21 on dist-test-slave-g31s"
server_key: "7c660dcaa781ab6ccce8b741a00514e6"
server_key_iv: "833eabc064a0dbd74845ad6e6a125718"
server_key_version: "encryptionkey@0"
I20260928 12:31:21.312431 16758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "d693500a9ae54bc988c9753c795142f6"
format_stamp: "Formatted at 2026-09-28 12:31:21 on dist-test-slave-g31s"
server_key: "7c660dcaa781ab6ccce8b741a00514e6"
server_key_iv: "833eabc064a0dbd74845ad6e6a125718"
server_key_version: "encryptionkey@0"
I20260928 12:31:21.313759 16758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:21.314625 16773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.314772 16758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:21.314855 16758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "d693500a9ae54bc988c9753c795142f6"
format_stamp: "Formatted at 2026-09-28 12:31:21 on dist-test-slave-g31s"
server_key: "7c660dcaa781ab6ccce8b741a00514e6"
server_key_iv: "833eabc064a0dbd74845ad6e6a125718"
server_key_version: "encryptionkey@0"
I20260928 12:31:21.314927 16758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:21.327921 16758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:21.330006 16758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:21.330147 16758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:21.330392 16758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:21.330675 16758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:21.330709 16758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.330739 16758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:21.330763 16758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.336666 16758 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:36879
I20260928 12:31:21.336716 16886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:36879 every 8 connection(s)
I20260928 12:31:21.337047 16758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598681, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:21.344549 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 16758
I20260928 12:31:21.344659 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:31:21.344853 8244 external_mini_cluster.cc:1428] Setting key 564c27e08dab8146e6c29d6b8a2f3ecc
I20260928 12:31:21.348753 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:21.337915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:38397 (local address 127.8.13.62:40511)
0928 12:31:21.337967 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:21.337969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:21.338142 (+ 173us) server_negotiation.cc:407] Connection header received
0928 12:31:21.338624 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:21.338625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:21.338635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:21.338698 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:21.339758 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.340340 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.341550 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.341777 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.341928 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.344813 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:21.344816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:21.344816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:21.344830 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:21.346652 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:21.347304 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:21.347306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:21.347306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:21.347394 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:21.347827 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:21.347828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:21.347828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:21.347975 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:21.348048 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:21.348581 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 12:31:21.348680 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:31:21.348989 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:21.337791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:38397)
0928 12:31:21.337983 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:21.338030 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:21.338541 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:21.338776 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:21.338780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:21.339384 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:21.339591 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.339597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.340492 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.340493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.341398 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.341401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.342000 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.342002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.342205 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.343015 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:21.343028 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:21.344616 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:21.346781 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:21.346799 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:21.346802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:21.347187 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:21.347526 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:21.347528 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:21.347528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:21.347723 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:21.348131 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:21.348140 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:21.348381 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 12:31:21.348497 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 12:31:21.348722 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260928 12:31:21.349648 16887 heartbeater.cc:347] Connected to a master server at 127.8.13.62:40511
I20260928 12:31:21.349779 16887 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:21.350020 16887 heartbeater.cc:511] Master 127.8.13.62:40511 requested a full tablet report, sending...
I20260928 12:31:21.350530 16428 ts_manager.cc:194] Registered new tserver with Master: d693500a9ae54bc988c9753c795142f6 (127.8.13.3:36879)
I20260928 12:31:21.350919 16428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:38397
I20260928 12:31:21.357723 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab.
Entry for principal oryx/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab.
WARNING: no policy specified for oryx/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab.
Entry for principal oryx/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab.
WARNING: no policy specified for oryx/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab.
Entry for principal oryx/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab.
I20260928 12:31:21.436858 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 16490
I20260928 12:31:21.441614 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 16624
I20260928 12:31:21.446040 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 16758
I20260928 12:31:21.451879 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 16396
I20260928 12:31:21.456980 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:40511
--webserver_interface=127.8.13.62
--webserver_port=38947
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:40511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:21.540117 16896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:21.540369 16896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:21.540395 16896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:21.541595 16896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:21.541646 16896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:21.541659 16896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:21.541671 16896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:21.541682 16896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:21.543304 16896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:40511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=38947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:21.543536 16896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:21.543723 16896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:21.548710 16901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:21.548722 16902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:21.548848 16896 server_base.cc:1034] running on GCE node
W20260928 12:31:21.548710 16904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:21.549201 16896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:21.549497 16896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:21.550689 16896 hybrid_clock.cc:648] HybridClock initialized: now 1790598681550663 us; error 37 us; skew 500 ppm
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598681, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:21.553319 16896 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:31:21.553776 16896 webserver.cc:460] Webserver started at http://127.8.13.62:38947/ using document root <none> and password file <none>
I20260928 12:31:21.554014 16896 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:21.554081 16896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:21.555694 16896 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:21.556618 16911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.556821 16896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:21.556918 16896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "b2aee5880a264b1cbf83523106b242a3"
format_stamp: "Formatted at 2026-09-28 12:31:20 on dist-test-slave-g31s"
server_key: "7a637791d9eff5a7f5b9e7b12a72227d"
server_key_iv: "bc0fe07c1599275467f38f68f57ccf66"
server_key_version: "encryptionkey@0"
I20260928 12:31:21.557395 16896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:21.566906 16896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:21.569577 16896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:21.569711 16896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:21.573644 16896 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:40511
I20260928 12:31:21.573710 16963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:40511 every 8 connection(s)
I20260928 12:31:21.574072 16896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:21.575604 16964 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:31:21.576242 16964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3: Bootstrap starting.
I20260928 12:31:21.582410 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 16896
I20260928 12:31:21.582754 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:36017
--local_ip_for_outbound_sockets=127.8.13.1
--tserver_master_addrs=127.8.13.62:40511
--webserver_port=42199
--webserver_interface=127.8.13.1
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.13.1
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:31:21.585171 16964 log.cc:824] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:21.587589 16964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:31:21.587924 16964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3: Bootstrap complete.
I20260928 12:31:21.590875 16964 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } }
I20260928 12:31:21.591173 16964 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2aee5880a264b1cbf83523106b242a3, State: Initialized, Role: FOLLOWER
I20260928 12:31:21.591357 16964 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } }
I20260928 12:31:21.591431 16964 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:21.591470 16964 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:21.591524 16964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:31:21.592875 16964 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } }
I20260928 12:31:21.593073 16964 leader_election.cc:302] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2aee5880a264b1cbf83523106b242a3; no voters:
I20260928 12:31:21.593361 16964 leader_election.cc:288] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:31:21.593428 16968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:31:21.593555 16968 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [term 2 LEADER]: Becoming Leader. State: Replica: b2aee5880a264b1cbf83523106b242a3, State: Running, Role: LEADER
I20260928 12:31:21.593700 16964 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:21.593664 16968 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } }
I20260928 12:31:21.594189 16969 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b2aee5880a264b1cbf83523106b242a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } } }
I20260928 12:31:21.594313 16969 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:21.594200 16970 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2aee5880a264b1cbf83523106b242a3. Latest consensus state: current_term: 2 leader_uuid: "b2aee5880a264b1cbf83523106b242a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aee5880a264b1cbf83523106b242a3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40511 } } }
I20260928 12:31:21.594441 16970 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:21.594559 16976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:21.595098 16976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:21.595242 16976 catalog_manager.cc:1259] Loaded cluster ID: 873ebe4849414f65b51e0d7a4ceb7b45
I20260928 12:31:21.595486 16976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:31:21.596779 16976 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:21.597004 16976 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b2aee5880a264b1cbf83523106b242a3: Loaded TSK: 0
I20260928 12:31:21.597237 16976 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:31:21.668445 16966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:21.668664 16966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:21.668691 16966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:21.670028 16966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:21.670079 16966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:21.670109 16966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:21.671731 16966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.1.keytab
--principal=oryx/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:36017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=42199
--tserver_master_addrs=127.8.13.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:21.671976 16966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:21.672211 16966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:21.677841 16991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:21.677856 16990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:21.677986 16993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:21.678223 16966 server_base.cc:1034] running on GCE node
I20260928 12:31:21.678448 16966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:21.678715 16966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:21.679888 16966 hybrid_clock.cc:648] HybridClock initialized: now 1790598681679809 us; error 96 us; skew 500 ppm
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598681, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:21.682583 16966 init.cc:377] Logged in from keytab as oryx/127.8.13.1@KRBTEST.COM (short username oryx)
I20260928 12:31:21.683097 16966 webserver.cc:460] Webserver started at http://127.8.13.1:42199/ using document root <none> and password file <none>
I20260928 12:31:21.683362 16966 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:21.683440 16966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:21.685055 16966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:31:21.685987 17000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.686183 16966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:31:21.686316 16966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "8c10ad764f4e41aca1397c8b88617d5b"
format_stamp: "Formatted at 2026-09-28 12:31:20 on dist-test-slave-g31s"
server_key: "b36f4d8d9d69b3c437ba1c14e48b8f9b"
server_key_iv: "7b415b3d7cbc09548547466d136cd81d"
server_key_version: "encryptionkey@0"
I20260928 12:31:21.686691 16966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:21.714211 16966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:21.716691 16966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:21.716864 16966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:21.717092 16966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:21.717422 16966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:21.717459 16966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.717490 16966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:21.717514 16966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.724437 16966 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:36017
I20260928 12:31:21.724464 17113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:36017 every 8 connection(s)
I20260928 12:31:21.724839 16966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:31:21.729029 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 16966
I20260928 12:31:21.729324 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:43531
--local_ip_for_outbound_sockets=127.8.13.2
--tserver_master_addrs=127.8.13.62:40511
--webserver_port=33165
--webserver_interface=127.8.13.2
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.13.2
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:21.735538 17116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:21.725596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:38975)
0928 12:31:21.725757 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:21.725835 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:21.726301 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:21.726753 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:21.726757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:21.727436 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:21.727631 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.727636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.729295 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.729297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.730529 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.730533 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.731440 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.731442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.731591 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.732453 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:21.732463 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:21.735336 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
W20260928 12:31:21.735762 17117 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:21.725657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:38975 (local address 127.8.13.62:40511)
0928 12:31:21.725864 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:21.725881 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:21.725921 (+ 40us) server_negotiation.cc:407] Connection header received
0928 12:31:21.726480 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:21.726486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:21.726541 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:21.726679 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:21.727735 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.729136 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.730638 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.731305 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.731430 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.735562 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:38975: BlockingRecv error: recv got EOF from 127.8.13.1:38975 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
W20260928 12:31:21.735800 17114 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:40511 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.13.62:40511: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:21.741669 17116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:21.736052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:53013)
0928 12:31:21.736168 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:21.736181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:21.736248 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:21.736386 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:21.736387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:21.736560 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:21.736632 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.736635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.737276 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.737278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.738162 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.738167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.738587 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.738588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.738666 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.739144 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:21.739150 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:21.741589 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260928 12:31:21.741779 17117 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:21.736116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:53013 (local address 127.8.13.62:40511)
0928 12:31:21.736170 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:21.736172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:21.736236 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:31:21.736314 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:21.736314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:21.736327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:21.736376 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:21.736696 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.737192 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.738285 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.738504 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.738571 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.741725 (+ 3154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:53013: BlockingRecv error: recv got EOF from 127.8.13.1:53013 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:21.747044 17116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:21.742022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:41825)
0928 12:31:21.742072 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:21.742083 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:21.742132 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:21.742268 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:21.742269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:21.742425 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:21.742476 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.742478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.743092 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.743093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.743952 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.743954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.744350 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.744351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.744408 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.744864 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:21.744870 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:21.746957 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:31:21.747151 17117 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:21.742071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:41825 (local address 127.8.13.62:40511)
0928 12:31:21.742116 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:21.742118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:21.742128 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:21.742194 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:21.742194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:21.742203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:21.742258 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:21.742539 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.743023 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.744066 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.744264 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.744342 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.747087 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:41825: BlockingRecv error: recv got EOF from 127.8.13.1:41825 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 12:31:21.747268 17114 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:31:21.819797 17118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:21.820042 17118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:21.820068 17118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:21.821405 17118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:21.821485 17118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:21.821520 17118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:21.823031 17118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.2.keytab
--principal=oryx/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:43531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=33165
--tserver_master_addrs=127.8.13.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:21.823246 17118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:21.823433 17118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:21.828409 17126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:21.828411 17124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:21.828459 17118 server_base.cc:1034] running on GCE node
W20260928 12:31:21.828409 17123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:21.828801 17118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:21.829056 17118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:21.830197 17118 hybrid_clock.cc:648] HybridClock initialized: now 1790598681830179 us; error 28 us; skew 500 ppm
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598681, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:21.832646 17118 init.cc:377] Logged in from keytab as oryx/127.8.13.2@KRBTEST.COM (short username oryx)
I20260928 12:31:21.833135 17118 webserver.cc:460] Webserver started at http://127.8.13.2:33165/ using document root <none> and password file <none>
I20260928 12:31:21.833338 17118 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:21.833405 17118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:21.834964 17118 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:21.835623 17133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.835868 17118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:21.835956 17118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "2cba36094dc1488eb596c34ad0bff674"
format_stamp: "Formatted at 2026-09-28 12:31:21 on dist-test-slave-g31s"
server_key: "cd8c369f98a9fc4237acd11137fdd7c5"
server_key_iv: "dc10599bc77d22aacd270d88fe9d13a3"
server_key_version: "encryptionkey@0"
I20260928 12:31:21.836339 17118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:21.852283 17118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:21.854843 17118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:21.855017 17118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:21.855261 17118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:21.855575 17118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:21.855607 17118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.855638 17118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:21.855659 17118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.861654 17118 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:43531
I20260928 12:31:21.861696 17246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:43531 every 8 connection(s)
I20260928 12:31:21.862099 17118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:31:21.865862 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 17118
I20260928 12:31:21.866149 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:36879
--local_ip_for_outbound_sockets=127.8.13.3
--tserver_master_addrs=127.8.13.62:40511
--webserver_port=37037
--webserver_interface=127.8.13.3
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.13.3
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:31:21.870368 17117 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:21.862965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:51217 (local address 127.8.13.62:40511)
0928 12:31:21.863054 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:21.863056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:21.863170 (+ 114us) server_negotiation.cc:407] Connection header received
0928 12:31:21.863682 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:21.863683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:21.863701 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:21.863787 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:21.864877 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.865396 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.866666 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.866951 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.867033 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.870301 (+ 3268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:51217: BlockingRecv error: recv got EOF from 127.8.13.2:51217 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 12:31:21.870409 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:21.862853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:51217)
0928 12:31:21.863019 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:21.863066 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:21.863571 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:21.863871 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:21.863874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:21.864486 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:21.864706 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.864711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.865513 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.865514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.866574 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.866577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.867072 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.867073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.867196 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.867686 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:21.867695 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:21.870155 (+ 2460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
W20260928 12:31:21.870635 17247 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:40511 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.13.62:40511: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:21.875064 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:21.870857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:49899)
0928 12:31:21.870920 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:21.870929 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:21.870967 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:21.871102 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:21.871103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:21.871242 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:21.871299 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.871302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.871940 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.871941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.872456 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.872458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.872798 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.872799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.872865 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.873294 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:21.873299 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:21.874988 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 12:31:21.875154 17117 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:21.870926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:49899 (local address 127.8.13.62:40511)
0928 12:31:21.870986 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:21.870988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:21.870998 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:21.871030 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:21.871030 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:21.871041 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:21.871090 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:21.871369 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.871868 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.872533 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.872736 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.872794 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.875097 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:49899: BlockingRecv error: recv got EOF from 127.8.13.2:49899 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:21.881053 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:21.875407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:45975)
0928 12:31:21.875469 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:21.875484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:21.875554 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:21.875692 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:21.875693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:21.875993 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:21.876076 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.876081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.876847 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.876848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.877833 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.877837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.878249 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.878250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:21.878316 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.878892 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:21.878901 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:21.880973 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 12:31:21.881150 17117 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:21.875457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:45975 (local address 127.8.13.62:40511)
0928 12:31:21.875492 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:21.875494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:21.875533 (+ 39us) server_negotiation.cc:407] Connection header received
0928 12:31:21.875618 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:21.875618 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:21.875627 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:21.875672 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:21.876242 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.876746 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.877942 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.878158 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.878236 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:21.881093 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:45975: BlockingRecv error: recv got EOF from 127.8.13.2:45975 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 12:31:21.881268 17247 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:31:21.953758 17250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:21.954046 17250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:21.954092 17250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:21.955406 17250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:21.955462 17250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:21.955492 17250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:21.957046 17250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/krb5kdc/oryx_127.8.13.3.keytab
--principal=oryx/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:36879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=37037
--tserver_master_addrs=127.8.13.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:21.957286 17250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:21.957518 17250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:21.962808 17255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:21.962844 17258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:21.962987 17250 server_base.cc:1034] running on GCE node
W20260928 12:31:21.962821 17256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:21.963253 17250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:21.963560 17250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:21.964727 17250 hybrid_clock.cc:648] HybridClock initialized: now 1790598681964712 us; error 29 us; skew 500 ppm
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598681, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:21.968909 17250 init.cc:377] Logged in from keytab as oryx/127.8.13.3@KRBTEST.COM (short username oryx)
I20260928 12:31:21.969686 17250 webserver.cc:460] Webserver started at http://127.8.13.3:37037/ using document root <none> and password file <none>
I20260928 12:31:21.970031 17250 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:21.970124 17250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:21.971827 17250 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:21.972954 17265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.973232 17250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:21.973399 17250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "d693500a9ae54bc988c9753c795142f6"
format_stamp: "Formatted at 2026-09-28 12:31:21 on dist-test-slave-g31s"
server_key: "7c660dcaa781ab6ccce8b741a00514e6"
server_key_iv: "833eabc064a0dbd74845ad6e6a125718"
server_key_version: "encryptionkey@0"
I20260928 12:31:21.974124 17250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:21.984407 17250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:21.987062 17250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:21.987221 17250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:21.987468 17250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:21.987741 17250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:21.987784 17250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.987831 17250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:21.987857 17250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:21.994849 17250 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:36879
I20260928 12:31:21.994884 17378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:36879 every 8 connection(s)
I20260928 12:31:21.995332 17250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:31:22.001888 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 17250
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:21 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:31:22.005673 17117 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:21.996100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:35585 (local address 127.8.13.62:40511)
0928 12:31:21.996161 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:21.996163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:21.996407 (+ 244us) server_negotiation.cc:407] Connection header received
0928 12:31:21.997147 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:21.997149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:21.997160 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:21.997217 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:21.998598 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.999352 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.000912 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.001230 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.001313 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:22.005606 (+ 4293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:35585: BlockingRecv error: recv got EOF from 127.8.13.3:35585 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:31:22.005966 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:21.996079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:35585)
0928 12:31:21.996260 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:21.996310 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:21.997011 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:21.997299 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:21.997304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:21.998147 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:21.998468 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:21.998476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:21.999494 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:21.999495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:22.000813 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:22.000817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.001337 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.001338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:22.001516 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:22.002772 (+ 1256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:22.002784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:22.005532 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
W20260928 12:31:22.006331 17379 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:40511 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.13.62:40511: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Sep 28 12:31:22 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:22 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:22.011842 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:22.002678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.8.13.62:40511)
0928 12:31:22.002801 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:22.002804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:22.002916 (+ 112us) server_negotiation.cc:407] Connection header received
0928 12:31:22.002999 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:22.003000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:22.003011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:22.003064 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:22.003730 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.004492 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.005578 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.005881 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.005947 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:22.007488 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:22.007494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:22.007495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:22.007510 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:22.009982 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:22.010398 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:22.010402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:22.010402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:22.010516 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:22.010693 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:22.010694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:22.010694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:22.010997 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:22.011123 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:22.011614 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 12:31:22.011728 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
W20260928 12:31:22.011996 17117 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:22.006597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:43225 (local address 127.8.13.62:40511)
0928 12:31:22.006635 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:22.006638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:22.006764 (+ 126us) server_negotiation.cc:407] Connection header received
0928 12:31:22.006827 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:22.006828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:22.006838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:22.006904 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:22.007241 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.007963 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.008989 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.009263 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.009301 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:22.011938 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:43225: BlockingRecv error: recv got EOF from 127.8.13.3:43225 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:31:22.012576 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:22.006563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:43225)
0928 12:31:22.006699 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:22.006710 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:22.006778 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:22.006923 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:22.006924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:22.007088 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:22.007152 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:22.007154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.008061 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.008062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:22.008894 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:22.008896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.009402 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.009403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:22.009464 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:22.009897 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:22.009901 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:22.012494 (+ 2593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
Sep 28 12:31:22 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:22 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:22.017869 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:22.012911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:47275)
0928 12:31:22.012951 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:22.012960 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:22.012992 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:22.013107 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:22.013108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:22.013248 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:22.013302 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:22.013304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.014194 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.014195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:22.014950 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:22.014953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.015413 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.015414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:22.015475 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:22.015909 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:22.015914 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:22.017761 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 12:31:22.017932 17383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:22.012975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:47275 (local address 127.8.13.62:40511)
0928 12:31:22.013011 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:22.013013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:22.013022 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:22.013041 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:22.013041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:22.013050 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:22.013102 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:22.013359 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.014108 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.015039 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.015315 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.015366 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:22.017879 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:47275: BlockingRecv error: recv got EOF from 127.8.13.3:47275 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 12:31:22.018096 17379 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 12:31:22 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:22 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:22.753690 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:22.747920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:34865)
0928 12:31:22.748079 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:22.748093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:22.748171 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:22.748620 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:22.748622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:22.748827 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:22.748895 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:22.748897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.749645 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.749646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:22.750333 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:22.750337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.750785 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.750787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:22.750838 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:22.751444 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:22.751451 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:22.753606 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
W20260928 12:31:22.753855 17386 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:22.748130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:34865 (local address 127.8.13.62:40511)
0928 12:31:22.748364 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:22.748366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:22.748378 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:31:22.748405 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:22.748406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:22.748418 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:22.748535 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:22.748991 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.749541 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.750450 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.750661 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.750729 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:22.753752 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:34865: BlockingRecv error: recv got EOF from 127.8.13.1:34865 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":166,"threads_started":1}
Sep 28 12:31:22 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:22 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:22.886550 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:22.881720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:40437)
0928 12:31:22.881925 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:22.881938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:22.882013 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:22.882159 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:22.882160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:22.882335 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:22.882387 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:22.882390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.883025 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.883026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:22.883577 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:22.883578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.883905 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.883906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:22.883962 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:22.884444 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:22.884448 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:22.886473 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
W20260928 12:31:22.886689 17386 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:22.881849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:40437 (local address 127.8.13.62:40511)
0928 12:31:22.881918 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:22.881920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:22.882002 (+ 82us) server_negotiation.cc:407] Connection header received
0928 12:31:22.882068 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:22.882069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:22.882082 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:22.882128 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:22.882458 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.882953 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.883652 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:22.883838 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:22.883893 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:22.886623 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:40437: BlockingRecv error: recv got EOF from 127.8.13.2:40437 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 12:31:23 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:23 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:23.023020 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:23.018517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:52203)
0928 12:31:23.018636 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:23.018648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:23.018711 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:23.018843 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:23.018844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:23.019017 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:23.019063 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:23.019065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.019714 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.019714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:23.020233 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:23.020235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.020564 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.020566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:23.020617 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:23.021101 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:23.021106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:23.022948 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20260928 12:31:23.023108 17386 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:23.018630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:52203 (local address 127.8.13.62:40511)
0928 12:31:23.018698 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:23.018700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:23.018710 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:23.018766 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:23.018767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:23.018780 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:23.018830 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:23.019136 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.019637 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.020321 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.020505 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.020560 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:23.023047 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:52203: BlockingRecv error: recv got EOF from 127.8.13.3:52203 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 12:31:23 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:23 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:23.759440 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:23.754183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:49085)
0928 12:31:23.754309 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:23.754321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:23.754368 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:23.754637 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:23.754638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:23.754812 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:23.754860 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:23.754861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.755563 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.755564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:23.756103 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:23.756105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.756494 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.756495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:23.756543 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:23.757131 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:23.757138 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:23.759369 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
W20260928 12:31:23.759614 17390 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:23.754234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:49085 (local address 127.8.13.62:40511)
0928 12:31:23.754346 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:23.754348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:23.754377 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:31:23.754429 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:23.754430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:23.754440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:23.754501 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:23.754933 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.755441 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.756202 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.756421 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.756482 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:23.759534 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:49085: BlockingRecv error: recv got EOF from 127.8.13.1:49085 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
Sep 28 12:31:23 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:23 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:23.892838 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:23.887067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:45339)
0928 12:31:23.887246 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:23.887257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:23.887321 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:23.887444 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:23.887445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:23.887606 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:23.887656 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:23.887657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.888485 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.888486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:23.889345 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:23.889349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.889964 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.889966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:23.890054 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:23.890787 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:23.890795 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:23.892770 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
W20260928 12:31:23.893044 17390 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:23.887101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:45339 (local address 127.8.13.62:40511)
0928 12:31:23.887159 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:23.887161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:23.887335 (+ 174us) server_negotiation.cc:407] Connection header received
0928 12:31:23.887364 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:23.887365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:23.887378 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:23.887433 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:23.887738 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.888331 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.889521 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:23.889785 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:23.889919 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:23.892974 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:45339: BlockingRecv error: recv got EOF from 127.8.13.2:45339 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 12:31:24 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:24 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:24.027751 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:24.023487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:45251)
0928 12:31:24.023628 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:24.023638 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:24.023694 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:24.023816 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:24.023817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:24.023969 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:24.024017 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:24.024019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.024601 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.024601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:24.025114 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:24.025115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.025385 (+ 270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.025385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:24.025422 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:24.025857 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:24.025862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:24.027680 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
W20260928 12:31:24.027863 17390 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:24.023612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:45251 (local address 127.8.13.62:40511)
0928 12:31:24.023654 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:24.023656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:24.023679 (+ 23us) server_negotiation.cc:407] Connection header received
0928 12:31:24.023748 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:24.023749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:24.023761 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:24.023806 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:24.024067 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.024549 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.025165 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.025337 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.025378 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:24.027798 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:45251: BlockingRecv error: recv got EOF from 127.8.13.3:45251 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 12:31:24 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:24 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:24.765975 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:24.760000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:47279)
0928 12:31:24.760165 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:24.760180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:24.760235 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:24.760656 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:24.760658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:24.760889 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:24.760962 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:24.760964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.761930 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.761932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:24.762564 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:24.762567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.762938 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.762939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:24.762998 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:24.763533 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:24.763537 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:24.765895 (+ 2358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
W20260928 12:31:24.766134 17394 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:24.760214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:47279 (local address 127.8.13.62:40511)
0928 12:31:24.760408 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:24.760410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:24.760423 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:31:24.760468 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:24.760469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:24.760485 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:24.760613 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:24.761143 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.761724 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.762667 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.762873 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.762930 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:24.766050 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:47279: BlockingRecv error: recv got EOF from 127.8.13.1:47279 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
Sep 28 12:31:24 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:24 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:24.898712 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:24.893437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:58897)
0928 12:31:24.893641 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:24.893653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:24.893731 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:24.893945 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:24.893946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:24.894106 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:24.894156 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:24.894158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.895054 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.895055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:24.895689 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:24.895691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.896074 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.896075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:24.896128 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:24.896594 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:24.896599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:24.898636 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":65,"threads_started":1}
W20260928 12:31:24.898828 17394 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:24.893609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:58897 (local address 127.8.13.62:40511)
0928 12:31:24.893679 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:24.893681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:24.893714 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:31:24.893825 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:24.893826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:24.893839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:24.893905 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:24.894248 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.894958 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.895788 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:24.895982 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:24.896039 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:24.898763 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:58897: BlockingRecv error: recv got EOF from 127.8.13.2:58897 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 12:31:25 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:25 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:25.032598 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:25.028304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:47043)
0928 12:31:25.028428 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:25.028442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:25.028525 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:25.028662 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:25.028663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:25.028823 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:25.028871 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:25.028873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.029498 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.029499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:25.030097 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:25.030100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.030458 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.030459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:25.030511 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:25.030931 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:25.030936 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:25.032520 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
W20260928 12:31:25.032670 17394 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:25.028402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:47043 (local address 127.8.13.62:40511)
0928 12:31:25.028460 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:25.028463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:25.028503 (+ 40us) server_negotiation.cc:407] Connection header received
0928 12:31:25.028588 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:25.028589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:25.028603 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:25.028651 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:25.028944 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.029426 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.030183 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.030400 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.030456 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:25.032612 (+ 2156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:47043: BlockingRecv error: recv got EOF from 127.8.13.3:47043 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 12:31:25 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:25 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:25.776192 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:25.766398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:40481)
0928 12:31:25.766559 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:25.766572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:25.766628 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:25.766757 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:25.766758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:25.767043 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:25.767127 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:25.767133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.767936 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.767937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:25.768552 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:25.768554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.769013 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.769014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:25.769095 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:25.774170 (+ 5075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:25.774178 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:25.776124 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
W20260928 12:31:25.776293 17398 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:25.766472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:40481 (local address 127.8.13.62:40511)
0928 12:31:25.766637 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:25.766638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:25.766647 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:25.766670 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:25.766671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:25.766681 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:25.766745 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:25.767284 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.767819 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.768674 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.768894 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.768983 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:25.776221 (+ 7238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:40481: BlockingRecv error: recv got EOF from 127.8.13.1:40481 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
Sep 28 12:31:25 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:25 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:25.904223 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:25.899324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:36737)
0928 12:31:25.899486 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:25.899499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:25.899585 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:25.899731 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:25.899732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:25.899914 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:25.899963 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:25.899965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.900666 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.900667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:25.901233 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:25.901236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.901626 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.901627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:25.901676 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:25.902254 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:25.902259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:25.904152 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
W20260928 12:31:25.904510 17398 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:25.899469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:36737 (local address 127.8.13.62:40511)
0928 12:31:25.899531 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:25.899533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:25.899566 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:31:25.899656 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:25.899657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:25.899669 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:25.899717 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:25.900044 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.900586 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.901339 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:25.901535 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:25.901597 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:25.904435 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:36737: BlockingRecv error: recv got EOF from 127.8.13.2:36737 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 12:31:26 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:26 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:26.038158 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:26.033118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:33899)
0928 12:31:26.033244 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:26.033255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:26.033324 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:26.033460 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:26.033462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:26.033650 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:26.033703 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:26.033705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.034399 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.034400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:26.034920 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:26.034922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.035573 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.035574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:26.035626 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:26.036088 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:26.036092 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:26.038094 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
W20260928 12:31:26.038354 17398 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:26.033176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:33899 (local address 127.8.13.62:40511)
0928 12:31:26.033246 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:26.033248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:26.033321 (+ 73us) server_negotiation.cc:407] Connection header received
0928 12:31:26.033376 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:26.033377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:26.033391 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:26.033439 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:26.033776 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.034296 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.035100 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.035353 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.035576 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:26.038275 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:33899: BlockingRecv error: recv got EOF from 127.8.13.3:33899 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 12:31:26 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:26 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:26.782996 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:26.776865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:45503)
0928 12:31:26.777001 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:26.777011 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:26.777065 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:26.777281 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:26.777282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:26.777450 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:26.777506 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:26.777507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.778377 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.778379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:26.778983 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:26.778985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.779599 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.779600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:26.779657 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:26.780104 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:26.780109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:26.782901 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
W20260928 12:31:26.783082 17402 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:26.777002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:45503 (local address 127.8.13.62:40511)
0928 12:31:26.777122 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:26.777124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:26.777133 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:26.777161 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:26.777162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:26.777171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:26.777270 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:26.777584 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.778261 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.779221 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.779476 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.779541 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:26.783017 (+ 3476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:45503: BlockingRecv error: recv got EOF from 127.8.13.1:45503 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
Sep 28 12:31:26 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:26 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:26.909273 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:26.904752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:44021)
0928 12:31:26.904882 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:26.904892 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:26.904943 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:26.905083 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:26.905083 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:26.905248 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:26.905305 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:26.905307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.905986 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.905987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:26.906505 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:26.906506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.906832 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.906833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:26.906887 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:26.907343 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:26.907347 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:26.909203 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
W20260928 12:31:26.909638 17402 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:26.904910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:44021 (local address 127.8.13.62:40511)
0928 12:31:26.904977 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:26.904979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:26.904987 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:26.905016 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:26.905016 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:26.905027 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:26.905072 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:26.905379 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.905908 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.906580 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:26.906767 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:26.906823 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:26.909547 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:44021: BlockingRecv error: recv got EOF from 127.8.13.2:44021 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 12:31:27 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:27 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:27.044292 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:27.038768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:58795)
0928 12:31:27.038948 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:27.038960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:27.039026 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:27.039189 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:27.039190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:27.039386 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:27.039437 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:27.039439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.040255 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.040257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:27.040831 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:27.040834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.041209 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.041210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:27.041264 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:27.041854 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:27.041861 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:27.044199 (+ 2338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":114,"threads_started":1}
W20260928 12:31:27.044389 17402 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:27.038908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:58795 (local address 127.8.13.62:40511)
0928 12:31:27.038968 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:27.038969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:27.039019 (+ 50us) server_negotiation.cc:407] Connection header received
0928 12:31:27.039097 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:27.039098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:27.039117 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:27.039189 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:27.039537 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.040138 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.040924 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.041111 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.041175 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:27.044315 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:58795: BlockingRecv error: recv got EOF from 127.8.13.3:58795 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 12:31:27 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:27 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:27.788596 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:27.783699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:44939)
0928 12:31:27.783876 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:27.783887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:27.783948 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:27.784240 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:27.784241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:27.784401 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:27.784452 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:27.784453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.785269 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.785270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:27.785831 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:27.785833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.786314 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.786315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:27.786367 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:27.786812 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:27.786817 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:27.788532 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":136,"threads_started":1}
W20260928 12:31:27.788774 17406 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:27.783887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:44939 (local address 127.8.13.62:40511)
0928 12:31:27.784083 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:27.784084 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:27.784093 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:27.784119 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:27.784119 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:27.784130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:27.784191 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:27.784522 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.785025 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.786016 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.786222 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.786291 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:27.788650 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:44939: BlockingRecv error: recv got EOF from 127.8.13.1:44939 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":128,"threads_started":1}
Sep 28 12:31:27 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:27 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:27.914189 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:27.909843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:48697)
0928 12:31:27.910006 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:27.910017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:27.910087 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:27.910236 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:27.910237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:27.910393 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:27.910440 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:27.910442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.911080 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.911081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:27.911639 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:27.911641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.911991 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.911992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:27.912045 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:27.912464 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:27.912467 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:27.914130 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
W20260928 12:31:27.914283 17406 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:27.909906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:48697 (local address 127.8.13.62:40511)
0928 12:31:27.909982 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:27.909985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:27.910071 (+ 86us) server_negotiation.cc:407] Connection header received
0928 12:31:27.910156 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:27.910157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:27.910167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:27.910208 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:27.910496 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.910987 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.911713 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:27.911900 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:27.911946 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:27.914223 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:48697: BlockingRecv error: recv got EOF from 127.8.13.2:48697 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 12:31:28 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:28 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:28.049312 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:28.044849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:60053)
0928 12:31:28.045017 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:28.045029 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:28.045103 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:28.045228 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:28.045229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:28.045409 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:28.045459 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:28.045461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.046165 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.046166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:28.046707 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:28.046709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.047016 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.047016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:28.047054 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:28.047501 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:28.047505 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:28.049238 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":74,"threads_started":1}
W20260928 12:31:28.049422 17406 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:28.044948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:60053 (local address 127.8.13.62:40511)
0928 12:31:28.045002 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:28.045005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:28.045098 (+ 93us) server_negotiation.cc:407] Connection header received
0928 12:31:28.045162 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:28.045162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:28.045175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:28.045213 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:28.045537 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.046102 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.046772 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.046965 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.047009 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:28.049356 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:60053: BlockingRecv error: recv got EOF from 127.8.13.3:60053 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 12:31:28 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:28 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:28.794528 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:28.789156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:55269)
0928 12:31:28.789287 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:28.789300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:28.789362 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:28.789670 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:28.789670 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:28.789865 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:28.789926 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:28.789973 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.790809 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.790810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:28.791385 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:28.791387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.791809 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.791810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:28.791865 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:28.792374 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:28.792379 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:28.794391 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
W20260928 12:31:28.794631 17410 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:28.789328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:55269 (local address 127.8.13.62:40511)
0928 12:31:28.789484 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:28.789486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:28.789498 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:31:28.789554 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:28.789555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:28.789568 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:28.789653 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:28.790117 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.790696 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.791497 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.791714 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.791781 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:28.794563 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:55269: BlockingRecv error: recv got EOF from 127.8.13.1:55269 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
Sep 28 12:31:28 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:28 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:28.919394 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:28.914727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:55583)
0928 12:31:28.914858 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:28.914868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:28.914930 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:28.915124 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:28.915125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:28.915297 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:28.915347 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:28.915348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.916040 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.916041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:28.916595 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:28.916597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.916921 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.916922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:28.916972 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:28.917458 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:28.917462 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:28.919250 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
W20260928 12:31:28.919423 17410 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:28.914840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:55583 (local address 127.8.13.62:40511)
0928 12:31:28.914910 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:28.914912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:28.914922 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:28.914985 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:28.914986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:28.914998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:28.915047 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:28.915447 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.915960 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.916661 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:28.916851 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:28.916898 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:28.919368 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:55583: BlockingRecv error: recv got EOF from 127.8.13.2:55583 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 12:31:29 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:29 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:29.055299 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:29.049966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:46071)
0928 12:31:29.050192 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:29.050208 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:29.050319 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:29.050550 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:29.050552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:29.050740 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:29.050841 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:29.050853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.051641 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.051644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:29.052194 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:29.052196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.052639 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.052640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:29.052701 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:29.053184 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:29.053190 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:29.055219 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":64,"threads_started":1}
W20260928 12:31:29.055365 17410 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:29.050163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:46071 (local address 127.8.13.62:40511)
0928 12:31:29.050233 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:29.050235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:29.050306 (+ 71us) server_negotiation.cc:407] Connection header received
0928 12:31:29.050382 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:29.050383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:29.050397 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:29.050469 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:29.050996 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.051513 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.052284 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.052557 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.052616 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:29.055308 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:46071: BlockingRecv error: recv got EOF from 127.8.13.3:46071 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 12:31:29 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:29 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:29.799820 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:29.795061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:53001)
0928 12:31:29.795201 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:29.795211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:29.795259 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:29.795523 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:29.795524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:29.795683 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:29.795739 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:29.795740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.796456 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.796457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:29.797005 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:29.797007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.797374 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.797375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:29.797413 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:29.797905 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:29.797911 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:29.799746 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
W20260928 12:31:29.799984 17414 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:29.795230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:53001 (local address 127.8.13.62:40511)
0928 12:31:29.795350 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:29.795352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:29.795360 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:29.795386 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:29.795387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:29.795396 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:29.795462 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:29.795823 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.796354 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.797114 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.797314 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.797369 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:29.799909 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:53001: BlockingRecv error: recv got EOF from 127.8.13.1:53001 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
Sep 28 12:31:29 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:29 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:29.924645 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:29.919954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:45065)
0928 12:31:29.920095 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:29.920106 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:29.920173 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:29.920308 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:29.920309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:29.920473 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:29.920517 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:29.920519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.921333 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.921334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:29.921939 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:29.921941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.922492 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.922493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:29.922543 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:29.923011 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:29.923016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:29.924588 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
W20260928 12:31:29.924779 17414 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:29.920029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:45065 (local address 127.8.13.62:40511)
0928 12:31:29.920081 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:29.920083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:29.920166 (+ 83us) server_negotiation.cc:407] Connection header received
0928 12:31:29.920238 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:29.920238 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:29.920250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:29.920293 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:29.920603 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.921243 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.922170 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:29.922386 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:29.922456 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:29.924709 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:45065: BlockingRecv error: recv got EOF from 127.8.13.2:45065 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 12:31:30 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:30 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:30.060871 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:30.055959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:46713)
0928 12:31:30.056106 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:30.056119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:30.056185 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:30.056321 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:30.056321 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:30.056479 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:30.056525 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:30.056527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.057315 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.057317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:30.057934 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:30.057937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.058318 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.058320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:30.058373 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:30.058848 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:30.058853 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:30.060795 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
W20260928 12:31:30.060978 17414 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:30.056065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:46713 (local address 127.8.13.62:40511)
0928 12:31:30.056162 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:30.056164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:30.056175 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:31:30.056240 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:30.056240 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:30.056253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:30.056305 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:30.056614 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.057253 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.058036 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.058228 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.058289 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:30.060927 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:46713: BlockingRecv error: recv got EOF from 127.8.13.3:46713 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 12:31:30 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:30 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:30.811203 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:30.800392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:53229)
0928 12:31:30.800543 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:30.800557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:30.800635 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:30.800811 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:30.800811 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:30.800974 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:30.801022 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:30.801023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.801834 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.801836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:30.802605 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:30.802610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.803012 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.803014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:30.803096 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:30.808642 (+ 5546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:30.808650 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:30.811122 (+ 2472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":58,"threads_started":1}
W20260928 12:31:30.811342 17418 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:30.800521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:53229 (local address 127.8.13.62:40511)
0928 12:31:30.800679 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:30.800681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:30.800692 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:31:30.800719 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:30.800719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:30.800730 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:30.800796 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:30.801113 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.801674 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.802693 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.802894 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.802942 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:30.811268 (+ 8326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:53229: BlockingRecv error: recv got EOF from 127.8.13.1:53229 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
Sep 28 12:31:30 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:30 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:30.931129 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:30.925299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:55933)
0928 12:31:30.925562 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:30.925575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:30.925658 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:30.925864 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:30.925865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:30.926077 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:30.926132 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:30.926134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.926841 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.926842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:30.927364 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:30.927366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.927783 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.927793 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:30.927880 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:30.928564 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:30.928570 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:30.930925 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
W20260928 12:31:30.931234 17418 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:30.925448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:55933 (local address 127.8.13.62:40511)
0928 12:31:30.925521 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:30.925523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:30.925666 (+ 143us) server_negotiation.cc:407] Connection header received
0928 12:31:30.925721 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:30.925722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:30.925735 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:30.925827 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:30.926240 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.926760 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.927461 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:30.927654 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:30.927720 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:30.931173 (+ 3453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:55933: BlockingRecv error: recv got EOF from 127.8.13.2:55933 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 12:31:31 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:31 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:31.067942 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:31.061474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:37949)
0928 12:31:31.061593 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:31.061603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:31.061652 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:31.061771 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:31.061772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:31.061951 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:31.062002 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:31.062004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.063040 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.063042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:31.063777 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:31.063784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.064296 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.064297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:31.064362 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:31.065123 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:31.065128 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:31.067873 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
W20260928 12:31:31.068109 17418 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:31.061575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:37949 (local address 127.8.13.62:40511)
0928 12:31:31.061627 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:31.061629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:31.061641 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:31:31.061699 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:31.061700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:31.061711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:31.061764 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:31.062156 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.062882 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.063913 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.064134 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.064248 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:31.068043 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:37949: BlockingRecv error: recv got EOF from 127.8.13.3:37949 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 12:31:31 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:31 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:31.816555 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:31.811822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:48107)
0928 12:31:31.811974 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:31.811984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:31.812035 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:31.812233 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:31.812234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:31.812414 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:31.812463 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:31.812464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.813206 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.813207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:31.813735 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:31.813736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.814096 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.814097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:31.814148 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:31.814594 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:31.814599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:31.816494 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
W20260928 12:31:31.816663 17422 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:31.811963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:48107 (local address 127.8.13.62:40511)
0928 12:31:31.812077 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:31.812079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:31.812088 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:31.812114 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:31.812115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:31.812125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:31.812199 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:31.812568 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.813097 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.813833 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.814030 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.814086 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:31.816593 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:48107: BlockingRecv error: recv got EOF from 127.8.13.1:48107 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
Sep 28 12:31:31 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:31 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:31.942353 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:31.931757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:51049)
0928 12:31:31.931926 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:31.931939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:31.932003 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:31.932153 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:31.932154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:31.932388 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:31.932444 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:31.932446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.933134 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.933135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:31.933709 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:31.933710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.934122 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.934123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:31.934168 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:31.940178 (+ 6010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:31.940185 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:31.942277 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
W20260928 12:31:31.942543 17422 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:31.931878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:51049 (local address 127.8.13.62:40511)
0928 12:31:31.931931 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:31.931933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:31.932001 (+ 68us) server_negotiation.cc:407] Connection header received
0928 12:31:31.932063 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:31.932064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:31.932077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:31.932136 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:31.932537 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.933049 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.933833 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:31.934053 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:31.934109 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:31.942433 (+ 8324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:51049: BlockingRecv error: recv got EOF from 127.8.13.2:51049 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 12:31:32 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:32 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:32.074165 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:32.068644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:47123)
0928 12:31:32.068846 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:32.068861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:32.068954 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:32.069118 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:32.069119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:32.069344 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:32.069414 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:32.069417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.070309 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.070310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:32.070886 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:32.070888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.071237 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.071238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:32.071302 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:32.072042 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:32.072049 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:32.074098 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
W20260928 12:31:32.074342 17422 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:32.068799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:47123 (local address 127.8.13.62:40511)
0928 12:31:32.068861 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:32.068863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:32.068929 (+ 66us) server_negotiation.cc:407] Connection header received
0928 12:31:32.069028 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:32.069029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:32.069047 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:32.069105 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:32.069494 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.070209 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.070971 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.071168 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.071234 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:32.074267 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:47123: BlockingRecv error: recv got EOF from 127.8.13.3:47123 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 12:31:32 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:32 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:32.822149 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:32.817094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:33771)
0928 12:31:32.817224 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:32.817234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:32.817284 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:32.817426 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:32.817427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:32.817582 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:32.817629 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:32.817630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.818339 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.818340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:32.818860 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:32.818862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.819296 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.819298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:32.819375 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:32.820274 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:32.820279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:32.822087 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
W20260928 12:31:32.822429 17426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:32.817169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:33771 (local address 127.8.13.62:40511)
0928 12:31:32.817304 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:32.817305 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:32.817315 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:32.817339 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:32.817339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:32.817350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:32.817416 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:32.817716 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.818269 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.818954 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.819153 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.819217 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:32.822307 (+ 3090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:33771: BlockingRecv error: recv got EOF from 127.8.13.1:33771 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
Sep 28 12:31:32 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:32 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:32.948637 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:32.942941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:49977)
0928 12:31:32.943101 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:32.943135 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:32.943187 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:32.943610 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:32.943611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:32.943773 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:32.943822 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:32.943824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.944604 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.944607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:32.945231 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:32.945233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.945754 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.945755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:32.945866 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:32.946449 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:32.946456 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:32.948573 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
W20260928 12:31:32.948931 17426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:32.943088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:49977 (local address 127.8.13.62:40511)
0928 12:31:32.943171 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:32.943173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:32.943183 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:32.943422 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:32.943422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:32.943435 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:32.943496 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:32.943934 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.944459 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.945374 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:32.945592 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:32.945663 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:32.948840 (+ 3177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:49977: BlockingRecv error: recv got EOF from 127.8.13.2:49977 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 12:31:33 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:33 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:33.080255 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:33.074905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:37023)
0928 12:31:33.075205 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:33.075221 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:33.075336 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:33.075699 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:33.075700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:33.075916 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:33.075991 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:33.075993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.076751 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.076753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:33.077345 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:33.077349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.077736 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.077737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:33.077793 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:33.078348 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:33.078352 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:33.080160 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":196,"threads_started":1}
W20260928 12:31:33.080327 17426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:33.075127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:37023 (local address 127.8.13.62:40511)
0928 12:31:33.075204 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:33.075207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:33.075309 (+ 102us) server_negotiation.cc:407] Connection header received
0928 12:31:33.075523 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:33.075529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:33.075565 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:33.075648 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:33.076112 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.076627 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.077444 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.077630 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.077687 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:33.080264 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:37023: BlockingRecv error: recv got EOF from 127.8.13.3:37023 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 12:31:33 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:33 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:33.827046 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:33.822674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:48975)
0928 12:31:33.822800 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:33.822810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:33.822859 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:33.823074 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:33.823075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:33.823235 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:33.823281 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:33.823283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.823959 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.823959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:33.824491 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:33.824493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.824799 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.824799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:33.824842 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:33.825300 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:33.825304 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:33.826973 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
W20260928 12:31:33.827170 17430 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:33.822802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:48975 (local address 127.8.13.62:40511)
0928 12:31:33.822914 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:33.822915 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:33.822924 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:31:33.822950 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:33.822951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:33.822961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:33.823029 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:33.823379 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.823888 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.824557 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.824741 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.824797 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:33.827096 (+ 2299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:48975: BlockingRecv error: recv got EOF from 127.8.13.1:48975 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":52,"threads_started":1}
Sep 28 12:31:33 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:33 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:33.954629 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:33.949394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:35851)
0928 12:31:33.949639 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:33.949652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:33.949770 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:33.950017 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:33.950019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:33.950278 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:33.950351 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:33.950354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.951150 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.951152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:33.951745 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:33.951747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.952102 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.952103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:33.952164 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:33.952630 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:33.952635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:33.954522 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":154,"threads_started":1}
W20260928 12:31:33.954694 17430 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:33.949516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:35851 (local address 127.8.13.62:40511)
0928 12:31:33.949579 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:33.949581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:33.949743 (+ 162us) server_negotiation.cc:407] Connection header received
0928 12:31:33.949902 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:33.949903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:33.949923 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:33.949989 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:33.950428 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.951023 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.951835 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:33.952041 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:33.952097 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:33.954637 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:35851: BlockingRecv error: recv got EOF from 127.8.13.2:35851 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 12:31:34 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:34 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:34.085917 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:34.080834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:48721)
0928 12:31:34.080959 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:34.080970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:34.081021 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:34.081158 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:34.081159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:34.081340 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:34.081391 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:34.081393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.082134 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.082135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:34.082670 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:34.082672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.083220 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.083221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:34.083275 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:34.083852 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:34.083857 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:34.085854 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
W20260928 12:31:34.086200 17430 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:34.080951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:48721 (local address 127.8.13.62:40511)
0928 12:31:34.081006 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:34.081007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:34.081018 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:31:34.081072 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:34.081073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:34.081085 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:34.081146 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:34.081487 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.082016 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.082833 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.083082 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.083192 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:34.086121 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:48721: BlockingRecv error: recv got EOF from 127.8.13.3:48721 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 12:31:34 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:34 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:34.833236 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:34.827700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:49843)
0928 12:31:34.827845 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:34.827858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:34.827936 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:34.828075 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:34.828076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:34.828281 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:34.828355 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:34.828358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.829034 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.829035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:34.829608 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:34.829611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.830236 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.830238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:34.830305 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:34.830923 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:34.830928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:34.833162 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
W20260928 12:31:34.833377 17434 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:34.827811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:49843 (local address 127.8.13.62:40511)
0928 12:31:34.827946 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:34.827948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:34.827956 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:34.827981 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:34.827982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:34.827994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:34.828063 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:34.828418 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.828934 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.829765 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.830062 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.830176 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:34.833293 (+ 3117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:49843: BlockingRecv error: recv got EOF from 127.8.13.1:49843 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
Sep 28 12:31:34 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:34 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:34.960081 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:34.955281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:49279)
0928 12:31:34.955519 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:34.955531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:34.955592 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:34.955728 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:34.955729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:34.955900 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:34.955952 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:34.955954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.956624 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.956625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:34.957152 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:34.957154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.957520 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.957521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:34.957559 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:34.958066 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:34.958071 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:34.960016 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":77,"threads_started":1}
W20260928 12:31:34.960260 17434 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:34.955493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:49279 (local address 127.8.13.62:40511)
0928 12:31:34.955569 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:34.955571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:34.955581 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:34.955658 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:34.955658 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:34.955670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:34.955720 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:34.956040 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.956557 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.957260 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:34.957460 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:34.957518 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:34.960187 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:49279: BlockingRecv error: recv got EOF from 127.8.13.2:49279 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 12:31:35 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:35 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:35.092435 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:35.086493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:57935)
0928 12:31:35.086757 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:35.086770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:35.086836 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:35.086955 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:35.086956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:35.087141 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:35.087189 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:35.087191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.088034 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.088036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:35.088689 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:35.088692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.089210 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.089212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:35.089269 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:35.089927 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:35.089933 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:35.092350 (+ 2417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":72,"threads_started":1}
W20260928 12:31:35.092523 17434 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:35.086578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:57935 (local address 127.8.13.62:40511)
0928 12:31:35.086626 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:35.086628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:35.086830 (+ 202us) server_negotiation.cc:407] Connection header received
0928 12:31:35.086879 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:35.086880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:35.086891 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:35.086940 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:35.087306 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.087890 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.088808 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.089116 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.089185 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:35.092457 (+ 3272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:57935: BlockingRecv error: recv got EOF from 127.8.13.3:57935 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 12:31:35 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:35 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:35.843431 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:35.833815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:40419)
0928 12:31:35.833936 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:35.833947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:35.834001 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:35.834234 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:35.834235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:35.834405 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:35.834464 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:35.834466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.835183 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.835184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:35.835737 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:35.835738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.836111 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.836113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:35.836179 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:35.841052 (+ 4873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:35.841059 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:35.843362 (+ 2303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
W20260928 12:31:35.843696 17438 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:35.833935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:40419 (local address 127.8.13.62:40511)
0928 12:31:35.834062 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:35.834064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:35.834072 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:35.834099 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:35.834100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:35.834110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:35.834187 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:35.834572 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.835106 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.835811 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.836052 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.836105 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:35.843613 (+ 7508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:40419: BlockingRecv error: recv got EOF from 127.8.13.1:40419 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
Sep 28 12:31:35 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:35 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:35.965446 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:35.960582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:51477)
0928 12:31:35.960727 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:35.960740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:35.960807 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:35.960952 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:35.960953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:35.961126 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:35.961174 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:35.961176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.961874 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.961876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:35.962416 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:35.962418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.962740 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.962741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:35.962792 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:35.963250 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:35.963255 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:35.965366 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
W20260928 12:31:35.965545 17438 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:35.960706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:51477 (local address 127.8.13.62:40511)
0928 12:31:35.960758 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:35.960760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:35.960789 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:31:35.960878 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:35.960879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:35.960893 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:35.960940 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:35.961242 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.961762 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.962495 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:35.962682 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:35.962737 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:35.965484 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:51477: BlockingRecv error: recv got EOF from 127.8.13.2:51477 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 12:31:36 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:36 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.3@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:36.098354 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:36.093086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.3:40653)
0928 12:31:36.093314 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:36.093329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:36.093411 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:36.093679 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:36.093680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:36.093921 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:36.093979 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:36.093981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.094958 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.094959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:36.095522 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:36.095524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.095912 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.095913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:36.095959 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:36.096397 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:36.096401 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:36.098273 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
W20260928 12:31:36.098511 17438 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:36.093200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:40653 (local address 127.8.13.62:40511)
0928 12:31:36.093265 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:36.093267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:36.093439 (+ 172us) server_negotiation.cc:407] Connection header received
0928 12:31:36.093570 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:36.093571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:36.093593 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:36.093661 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:36.094091 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.094877 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.095626 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.095830 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.095894 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:36.098433 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.3:40653: BlockingRecv error: recv got EOF from 127.8.13.3:40653 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 12:31:36 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:36 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.1@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:36.849488 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:36.843942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.1:42431)
0928 12:31:36.844068 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:36.844079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:36.844149 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:36.844289 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:36.844289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:36.844444 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:36.844494 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:36.844496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.845149 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.845150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:36.845745 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:36.845746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.846181 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.846181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:36.846231 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:36.846922 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:36.846930 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:36.849382 (+ 2452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
W20260928 12:31:36.849597 17442 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:36.844063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:42431 (local address 127.8.13.62:40511)
0928 12:31:36.844174 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:36.844175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:36.844183 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:36.844205 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:36.844206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:36.844215 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:36.844280 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:36.844542 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.845062 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.845889 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.846094 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.846150 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:36.849530 (+ 3380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.1:42431: BlockingRecv error: recv got EOF from 127.8.13.1:42431 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
Sep 28 12:31:36 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:31:36 dist-test-slave-g31s krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.2@KRBTEST.COM for oryx/127.8.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:31:36.972411 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:36.966244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40511 (local address 127.8.13.2:47273)
0928 12:31:36.966426 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:36.966440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:36.966519 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:36.966739 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:36.966747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:36.966931 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:36.966985 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:36.966987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.967793 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.967795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:36.968555 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:36.968558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.969121 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.969123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:36.969209 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:36.970105 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:36.970111 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:36.972331 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.62:40511: Server oryx/127.8.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
W20260928 12:31:36.972622 17442 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:36.966463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:47273 (local address 127.8.13.62:40511)
0928 12:31:36.966537 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:36.966539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:36.966553 (+ 14us) server_negotiation.cc:407] Connection header received
0928 12:31:36.966593 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:36.966594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:36.966607 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:36.966684 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:36.967125 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.967661 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.968650 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:36.968944 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:36.969120 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:36.972542 (+ 3422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.2:47273: BlockingRecv error: recv got EOF from 127.8.13.2:47273 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:31:37.012964 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 16966
I20260928 12:31:37.017370 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 17118
I20260928 12:31:37.021550 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 17250
I20260928 12:31:37.025441 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 16896
2026-09-28T12:31:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17439 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T12:31:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:37Z Disabled control of system clock
I20260928 12:31:37.053079 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:44483
--webserver_interface=127.8.13.62
--webserver_port=0
--builtin_ntp_servers=127.8.13.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:44483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 12:31:37.142702 17447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:37.142982 17447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:37.143013 17447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:37.144502 17447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:37.144569 17447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:37.144586 17447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:37.144603 17447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:37.144618 17447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:37.146526 17447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:44483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:44483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:37.146963 17447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:37.147392 17447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:37.152801 17452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:37.152832 17455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:37.152828 17453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:37.152889 17447 server_base.cc:1034] running on GCE node
I20260928 12:31:37.153368 17447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:37.153728 17447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:37.155063 17447 hybrid_clock.cc:648] HybridClock initialized: now 1790598697155039 us; error 38 us; skew 500 ppm
I20260928 12:31:37.156539 17447 webserver.cc:460] Webserver started at http://127.8.13.62:38349/ using document root <none> and password file <none>
I20260928 12:31:37.156802 17447 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:37.156855 17447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:37.157099 17447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:37.158319 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "facd6efe54ce41f6834e71067da3d49e"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "7cfdf0ca4e2390a84417175a197cb1ae"
server_key_iv: "b8bd9ec99cb4592d315565471f633f8b"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.158674 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "facd6efe54ce41f6834e71067da3d49e"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "7cfdf0ca4e2390a84417175a197cb1ae"
server_key_iv: "b8bd9ec99cb4592d315565471f633f8b"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.160249 17447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:37.161325 17461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.161540 17447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:37.161623 17447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "facd6efe54ce41f6834e71067da3d49e"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "7cfdf0ca4e2390a84417175a197cb1ae"
server_key_iv: "b8bd9ec99cb4592d315565471f633f8b"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.161695 17447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:37.177886 17447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:37.178241 17447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:37.178375 17447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:37.182812 17447 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:44483
I20260928 12:31:37.182957 17513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:44483 every 8 connection(s)
I20260928 12:31:37.183357 17447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:37.184224 17514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:37.187191 17514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e: Bootstrap starting.
I20260928 12:31:37.187793 17514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:37.188040 17514 log.cc:824] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:37.188632 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 17447
I20260928 12:31:37.188741 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:37.188897 17514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e: No bootstrap required, opened a new log
I20260928 12:31:37.188930 8244 external_mini_cluster.cc:1428] Setting key 56d7dae06409ba826e3d3d7033569b84
I20260928 12:31:37.190980 17514 raft_consensus.cc:348] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facd6efe54ce41f6834e71067da3d49e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44483 } }
I20260928 12:31:37.191166 17514 raft_consensus.cc:374] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:37.191193 17514 raft_consensus.cc:729] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facd6efe54ce41f6834e71067da3d49e, State: Initialized, Role: FOLLOWER
I20260928 12:31:37.191373 17514 consensus_queue.cc:260] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facd6efe54ce41f6834e71067da3d49e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44483 } }
I20260928 12:31:37.191462 17514 raft_consensus.cc:388] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:37.191504 17514 raft_consensus.cc:482] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:37.191556 17514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:37.192173 17514 raft_consensus.cc:504] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facd6efe54ce41f6834e71067da3d49e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44483 } }
I20260928 12:31:37.192283 17514 leader_election.cc:302] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: facd6efe54ce41f6834e71067da3d49e; no voters:
I20260928 12:31:37.192452 17514 leader_election.cc:288] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:37.192515 17519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:37.192716 17514 sys_catalog.cc:564] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:37.192736 17519 raft_consensus.cc:686] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [term 1 LEADER]: Becoming Leader. State: Replica: facd6efe54ce41f6834e71067da3d49e, State: Running, Role: LEADER
I20260928 12:31:37.192831 17519 consensus_queue.cc:237] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facd6efe54ce41f6834e71067da3d49e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44483 } }
I20260928 12:31:37.193212 17519 sys_catalog.cc:455] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [sys.catalog]: SysCatalogTable state changed. Reason: New leader facd6efe54ce41f6834e71067da3d49e. Latest consensus state: current_term: 1 leader_uuid: "facd6efe54ce41f6834e71067da3d49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facd6efe54ce41f6834e71067da3d49e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44483 } } }
I20260928 12:31:37.193286 17519 sys_catalog.cc:458] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:37.193951 17527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:37.194475 17527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:37.194490 17520 sys_catalog.cc:455] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "facd6efe54ce41f6834e71067da3d49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facd6efe54ce41f6834e71067da3d49e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44483 } } }
I20260928 12:31:37.194621 17520 sys_catalog.cc:458] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:37.196501 17527 catalog_manager.cc:1347] Generated new cluster ID: cc3dc90b66734cf9ae321741f98c10a1
I20260928 12:31:37.196580 17527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:31:37.196681 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:37.189768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39306 (local address 127.8.13.62:44483)
0928 12:31:37.190039 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:37.190050 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:37.190099 (+ 49us) server_negotiation.cc:407] Connection header received
0928 12:31:37.190969 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:37.190982 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:37.191069 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:37.191243 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:37.191501 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.193020 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.194205 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.194834 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.194971 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:37.195727 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:37.195731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:37.195732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:37.195947 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:37.195993 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:37.196009 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:37.196071 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:37.196304 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 12:31:37.196403 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":126,"threads_started":1}
I20260928 12:31:37.217053 17527 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:37.217583 17527 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:37.224733 17527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P facd6efe54ce41f6834e71067da3d49e: Generated new TSK 0
I20260928 12:31:37.224921 17527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:31:37.232589 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44483
--builtin_ntp_servers=127.8.13.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 12:31:37.321532 17538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:37.321744 17538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:37.321769 17538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:37.323041 17538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:37.323093 17538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:37.323118 17538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:37.324474 17538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:37.324712 17538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:37.324999 17538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:37.329718 17546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:37.329780 17543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:37.329836 17538 server_base.cc:1034] running on GCE node
W20260928 12:31:37.329713 17544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:37.330179 17538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:37.330437 17538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:37.331570 17538 hybrid_clock.cc:648] HybridClock initialized: now 1790598697331563 us; error 33 us; skew 500 ppm
I20260928 12:31:37.332844 17538 webserver.cc:460] Webserver started at http://127.8.13.1:39097/ using document root <none> and password file <none>
I20260928 12:31:37.333057 17538 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:37.333105 17538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:37.333230 17538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:37.334338 17538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "8fe3b8b9a4464b33837e4cd971e7f281"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "69d6344d7e0a8da39208b9ca2bb85c1d"
server_key_iv: "747b4dca6bc038496c1139a4a88e40cd"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.334668 17538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "8fe3b8b9a4464b33837e4cd971e7f281"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "69d6344d7e0a8da39208b9ca2bb85c1d"
server_key_iv: "747b4dca6bc038496c1139a4a88e40cd"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.336067 17538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:37.336930 17552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.337134 17538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:37.337249 17538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "8fe3b8b9a4464b33837e4cd971e7f281"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "69d6344d7e0a8da39208b9ca2bb85c1d"
server_key_iv: "747b4dca6bc038496c1139a4a88e40cd"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.337317 17538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:37.361582 17538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:37.362018 17538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:37.362181 17538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:37.362432 17538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:37.362804 17538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:37.362840 17538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.362870 17538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:37.362887 17538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.369282 17538 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:34581
I20260928 12:31:37.369351 17665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:34581 every 8 connection(s)
I20260928 12:31:37.369688 17538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:31:37.375914 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:37.370853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:58303 (local address 127.8.13.62:44483)
0928 12:31:37.370916 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:37.370918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:37.371357 (+ 439us) server_negotiation.cc:407] Connection header received
0928 12:31:37.371953 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:37.371954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:37.371971 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:37.372022 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:37.372802 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.373309 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.374361 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.374560 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.374620 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:37.375495 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:37.375497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:37.375497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:37.375502 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:37.375519 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:37.375523 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:37.375565 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:37.375652 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 12:31:37.375783 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:31:37.376250 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:37.370817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:44483 (local address 127.8.13.1:58303)
0928 12:31:37.371169 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:37.371224 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:37.371837 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:37.372103 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:37.372107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:37.372120 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:37.372705 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:37.372712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.373400 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.373401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:37.374272 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:37.374276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.374646 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.374647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:37.374781 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:37.375347 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:37.375347 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:37.375381 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:37.375409 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:37.375574 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:37.375579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:37.375582 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:37.375653 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:31:37.375943 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":133,"threads_started":1}
I20260928 12:31:37.377056 17666 heartbeater.cc:347] Connected to a master server at 127.8.13.62:44483
I20260928 12:31:37.377189 17666 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:37.377404 17666 heartbeater.cc:511] Master 127.8.13.62:44483 requested a full tablet report, sending...
I20260928 12:31:37.378044 17478 ts_manager.cc:194] Registered new tserver with Master: 8fe3b8b9a4464b33837e4cd971e7f281 (127.8.13.1:34581)
I20260928 12:31:37.378407 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 17538
I20260928 12:31:37.378485 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:31:37.378693 8244 external_mini_cluster.cc:1428] Setting key 43fc1e675420a789b82293e001927637
I20260928 12:31:37.378736 17478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.1:58303
I20260928 12:31:37.379712 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44483
--builtin_ntp_servers=127.8.13.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 12:31:37.462142 17669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:37.462481 17669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:37.462512 17669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:37.465878 17669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:37.465965 17669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:37.466008 17669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:37.468307 17669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:37.468631 17669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:37.468971 17669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:31:37.474421 17669 server_base.cc:1034] running on GCE node
W20260928 12:31:37.474421 17675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:37.474541 17674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:37.474541 17677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:37.474936 17669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:37.475241 17669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:37.476425 17669 hybrid_clock.cc:648] HybridClock initialized: now 1790598697476406 us; error 39 us; skew 500 ppm
I20260928 12:31:37.478170 17669 webserver.cc:460] Webserver started at http://127.8.13.2:36753/ using document root <none> and password file <none>
I20260928 12:31:37.478442 17669 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:37.478498 17669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:37.478624 17669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:37.479835 17669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "2064c47eba7448a69402558243e34174"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "eba888930eff85107735a53bbaab0b3d"
server_key_iv: "11b21083aa0db78f1eb808b2af166ddf"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.480206 17669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "2064c47eba7448a69402558243e34174"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "eba888930eff85107735a53bbaab0b3d"
server_key_iv: "11b21083aa0db78f1eb808b2af166ddf"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.481541 17669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:31:37.482431 17683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.482721 17669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:31:37.482816 17669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "2064c47eba7448a69402558243e34174"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "eba888930eff85107735a53bbaab0b3d"
server_key_iv: "11b21083aa0db78f1eb808b2af166ddf"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.482892 17669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:37.494828 17669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:37.495167 17669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:37.495313 17669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:37.495537 17669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:37.495857 17669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:37.495890 17669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.495920 17669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:37.495944 17669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.502305 17669 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:44453
I20260928 12:31:37.502353 17796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:44453 every 8 connection(s)
I20260928 12:31:37.502694 17669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:31:37.505388 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 17669
I20260928 12:31:37.505494 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:31:37.505692 8244 external_mini_cluster.cc:1428] Setting key c182a2b924d5af3a5d1f8f1190812117
I20260928 12:31:37.506676 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44483
--builtin_ntp_servers=127.8.13.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 12:31:37.508459 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:37.503488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:59339 (local address 127.8.13.62:44483)
0928 12:31:37.503557 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:37.503559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:37.503840 (+ 281us) server_negotiation.cc:407] Connection header received
0928 12:31:37.504363 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:37.504364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:37.504375 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:37.504431 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:37.505112 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.505904 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.507067 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.507334 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.507407 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:37.508178 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:37.508179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:37.508180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:37.508185 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:37.508202 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:37.508213 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:37.508273 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:37.508342 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 12:31:37.508373 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:31:37.508621 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:37.503458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:44483 (local address 127.8.13.2:59339)
0928 12:31:37.503648 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:37.503690 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:37.504230 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:37.504485 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:37.504489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:37.504498 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:37.505003 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:37.505006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.506015 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.506016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:37.506987 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:37.506991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.507413 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.507414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:37.507525 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:37.508040 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:37.508040 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:37.508071 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:37.508096 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:37.508278 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:37.508281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:37.508283 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:31:37.508331 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:31:37.508420 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260928 12:31:37.509341 17797 heartbeater.cc:347] Connected to a master server at 127.8.13.62:44483
I20260928 12:31:37.509447 17797 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:37.509642 17797 heartbeater.cc:511] Master 127.8.13.62:44483 requested a full tablet report, sending...
I20260928 12:31:37.510028 17478 ts_manager.cc:194] Registered new tserver with Master: 2064c47eba7448a69402558243e34174 (127.8.13.2:44453)
I20260928 12:31:37.510582 17478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.2:59339
W20260928 12:31:37.591136 17800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:37.591392 17800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:37.591425 17800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:37.592749 17800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:37.592804 17800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:37.592834 17800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:37.594517 17800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:37.594769 17800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:37.595063 17800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:31:37.600538 17800 server_base.cc:1034] running on GCE node
W20260928 12:31:37.600474 17808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:37.600474 17805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:37.600474 17806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:37.601040 17800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:37.601343 17800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:37.602516 17800 hybrid_clock.cc:648] HybridClock initialized: now 1790598697602492 us; error 40 us; skew 500 ppm
I20260928 12:31:37.604048 17800 webserver.cc:460] Webserver started at http://127.8.13.3:37521/ using document root <none> and password file <none>
I20260928 12:31:37.604314 17800 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:37.604370 17800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:37.604501 17800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:37.606389 17800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "68dfd8fe0e094339aa14923ea258da70"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "cd6ebdd0fdc5fb9a7daebb369b30ff57"
server_key_iv: "0d1e781f840f760587c61750f04124fc"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.606992 17800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "68dfd8fe0e094339aa14923ea258da70"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "cd6ebdd0fdc5fb9a7daebb369b30ff57"
server_key_iv: "0d1e781f840f760587c61750f04124fc"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.608783 17800 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 12:31:37.610248 17814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.610564 17800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:37.610674 17800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "68dfd8fe0e094339aa14923ea258da70"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "cd6ebdd0fdc5fb9a7daebb369b30ff57"
server_key_iv: "0d1e781f840f760587c61750f04124fc"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.610782 17800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:37.633946 17800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:37.634315 17800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:37.634455 17800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:37.634717 17800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:37.635075 17800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:37.635114 17800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.635147 17800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:37.635164 17800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.641507 17800 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:36207
I20260928 12:31:37.641562 17927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:36207 every 8 connection(s)
I20260928 12:31:37.641976 17800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:31:37.642349 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 17800
I20260928 12:31:37.642427 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:31:37.642614 8244 external_mini_cluster.cc:1428] Setting key e74497fad7efd1b05784911cb11ad57d
I20260928 12:31:37.647701 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:37.642709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:48215 (local address 127.8.13.62:44483)
0928 12:31:37.642760 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:37.642762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:37.643046 (+ 284us) server_negotiation.cc:407] Connection header received
0928 12:31:37.643708 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:37.643710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:37.643723 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:37.643761 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:37.644521 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.645137 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.646286 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.646502 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.646558 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:37.647422 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:37.647423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:37.647424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:37.647429 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:37.647445 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:37.647449 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:37.647491 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:37.647565 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 12:31:37.647597 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:31:37.647900 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:37.642721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:44483 (local address 127.8.13.3:48215)
0928 12:31:37.642911 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:37.642957 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:37.643610 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:37.643847 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:37.643852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:37.643861 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:37.644437 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:37.644443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.645267 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.645268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:37.646203 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:37.646206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:37.646578 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:37.646579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:37.646717 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:37.647287 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:37.647287 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:37.647318 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:37.647343 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:37.647496 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:37.647501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:37.647504 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:37.647565 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 12:31:37.647679 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260928 12:31:37.648578 17928 heartbeater.cc:347] Connected to a master server at 127.8.13.62:44483
I20260928 12:31:37.648689 17928 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:37.648857 17928 heartbeater.cc:511] Master 127.8.13.62:44483 requested a full tablet report, sending...
I20260928 12:31:37.649232 17478 ts_manager.cc:194] Registered new tserver with Master: 68dfd8fe0e094339aa14923ea258da70 (127.8.13.3:36207)
I20260928 12:31:37.649580 17478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.3:48215
I20260928 12:31:37.656070 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:31:37.658028 17518 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:37.657515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.8.13.62:44483)
0928 12:31:37.657588 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:37.657590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:37.657693 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:31:37.657725 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:37.657726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:37.657739 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:37.657827 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:37.657947 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39318: BlockingRecv error: recv got EOF from 127.0.0.1:39318 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:31:37.658077 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 17538
I20260928 12:31:37.662588 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 17669
I20260928 12:31:37.666695 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 17800
I20260928 12:31:37.671214 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 17447
2026-09-28T12:31:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (646 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T12:31:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:37Z Disabled control of system clock
I20260928 12:31:37.700222 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:44215
--webserver_interface=127.8.13.62
--webserver_port=0
--builtin_ntp_servers=127.8.13.20:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:44215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:31:37.788295 17939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:37.788530 17939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:37.788553 17939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:37.790154 17939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:37.790236 17939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:37.790251 17939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:37.790262 17939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:37.790273 17939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:37.791788 17939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:44215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:44215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:37.792040 17939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:37.792315 17939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:37.797459 17947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:37.797467 17944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:37.797508 17939 server_base.cc:1034] running on GCE node
W20260928 12:31:37.798086 17945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:37.798398 17939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:37.798755 17939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:37.799952 17939 hybrid_clock.cc:648] HybridClock initialized: now 1790598697799931 us; error 37 us; skew 500 ppm
I20260928 12:31:37.801359 17939 webserver.cc:460] Webserver started at http://127.8.13.62:35619/ using document root <none> and password file <none>
I20260928 12:31:37.801594 17939 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:37.801644 17939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:37.801751 17939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:37.804087 17939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "933fb35881ee4f1baa6725aa16bf3dca"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "df5688048e7f08e74f8056b23f3d9b14"
server_key_iv: "437a2713774781d9e44a871a0f9e22f6"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.804630 17939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "933fb35881ee4f1baa6725aa16bf3dca"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "df5688048e7f08e74f8056b23f3d9b14"
server_key_iv: "437a2713774781d9e44a871a0f9e22f6"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.808127 17939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 12:31:37.809034 17953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.809249 17939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:37.809322 17939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "933fb35881ee4f1baa6725aa16bf3dca"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "df5688048e7f08e74f8056b23f3d9b14"
server_key_iv: "437a2713774781d9e44a871a0f9e22f6"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.809394 17939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:37.813424 17939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:37.813722 17939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:37.813872 17939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:37.817711 17939 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:44215
I20260928 12:31:37.817853 18005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:44215 every 8 connection(s)
I20260928 12:31:37.818166 17939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:37.819021 18006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:37.822242 18006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca: Bootstrap starting.
I20260928 12:31:37.822841 18006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:37.823109 18006 log.cc:824] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:37.823797 18006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca: No bootstrap required, opened a new log
I20260928 12:31:37.825426 18006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933fb35881ee4f1baa6725aa16bf3dca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44215 } }
I20260928 12:31:37.825579 18006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:37.825608 18006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933fb35881ee4f1baa6725aa16bf3dca, State: Initialized, Role: FOLLOWER
I20260928 12:31:37.825755 18006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933fb35881ee4f1baa6725aa16bf3dca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44215 } }
I20260928 12:31:37.825824 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 17939
I20260928 12:31:37.825876 18006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:37.825906 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:37.825918 18006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:37.825961 18006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:37.826097 8244 external_mini_cluster.cc:1428] Setting key f57ca22ea45522cd65aa7c981517b13e
I20260928 12:31:37.826684 18006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933fb35881ee4f1baa6725aa16bf3dca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44215 } }
I20260928 12:31:37.826799 18006 leader_election.cc:302] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 933fb35881ee4f1baa6725aa16bf3dca; no voters:
I20260928 12:31:37.826952 18006 leader_election.cc:288] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:37.826983 18011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:37.827155 18011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [term 1 LEADER]: Becoming Leader. State: Replica: 933fb35881ee4f1baa6725aa16bf3dca, State: Running, Role: LEADER
I20260928 12:31:37.827198 18006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:37.827246 18011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933fb35881ee4f1baa6725aa16bf3dca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44215 } }
I20260928 12:31:37.827564 18011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "933fb35881ee4f1baa6725aa16bf3dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933fb35881ee4f1baa6725aa16bf3dca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44215 } } }
I20260928 12:31:37.827644 18011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:37.827756 18011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 933fb35881ee4f1baa6725aa16bf3dca. Latest consensus state: current_term: 1 leader_uuid: "933fb35881ee4f1baa6725aa16bf3dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933fb35881ee4f1baa6725aa16bf3dca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44215 } } }
I20260928 12:31:37.827804 18011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:37.827914 18017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:37.828610 18017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:37.828704 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:37.826592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.8.13.62:44215)
0928 12:31:37.826807 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:37.826817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:37.826850 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:31:37.826977 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:37.826984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:37.827031 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:37.827188 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:37.828140 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:37.828144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:37.828145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:37.828149 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:37.828198 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:37.828212 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:37.828400 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:37.828432 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 12:31:37.828497 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260928 12:31:37.830138 18017 catalog_manager.cc:1347] Generated new cluster ID: 2d6a70cb09b643678048fa17b011584e
I20260928 12:31:37.830194 18017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:31:37.850049 18017 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:37.850267 18017 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:37.857380 18017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 933fb35881ee4f1baa6725aa16bf3dca: Generated new TSK 0
I20260928 12:31:37.857599 18017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:31:37.860900 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44215
--builtin_ntp_servers=127.8.13.20:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:31:37.958115 18030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:37.958367 18030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:37.958403 18030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:37.959861 18030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:37.959939 18030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:37.959983 18030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:37.961639 18030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:37.962242 18030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:37.962882 18030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:37.969074 18036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:37.969075 18035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:37.969103 18038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:37.969841 18030 server_base.cc:1034] running on GCE node
I20260928 12:31:37.970096 18030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:37.970363 18030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:37.971546 18030 hybrid_clock.cc:648] HybridClock initialized: now 1790598697971550 us; error 76 us; skew 500 ppm
I20260928 12:31:37.972956 18030 webserver.cc:460] Webserver started at http://127.8.13.1:36483/ using document root <none> and password file <none>
I20260928 12:31:37.973204 18030 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:37.973264 18030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:37.973385 18030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:37.974609 18030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "880da60384f64078bd5ab8d2ae6daa25"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "6be53e78b045d340b883bfbd2d4166ce"
server_key_iv: "4f4ef61afaca7d282e6fdab5cd86ef1f"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.974951 18030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "880da60384f64078bd5ab8d2ae6daa25"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "6be53e78b045d340b883bfbd2d4166ce"
server_key_iv: "4f4ef61afaca7d282e6fdab5cd86ef1f"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.976361 18030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:31:37.977428 18044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.977669 18030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:37.977763 18030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "880da60384f64078bd5ab8d2ae6daa25"
format_stamp: "Formatted at 2026-09-28 12:31:37 on dist-test-slave-g31s"
server_key: "6be53e78b045d340b883bfbd2d4166ce"
server_key_iv: "4f4ef61afaca7d282e6fdab5cd86ef1f"
server_key_version: "encryptionkey@0"
I20260928 12:31:37.977877 18030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:37.981495 18030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:37.981830 18030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:37.981974 18030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:37.982173 18030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:37.982513 18030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:37.982548 18030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.982577 18030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:37.982602 18030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:37.990450 18030 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:39269
I20260928 12:31:37.990545 18157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:39269 every 8 connection(s)
I20260928 12:31:37.990869 18030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:31:37.994879 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:37.991606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:41557 (local address 127.8.13.62:44215)
0928 12:31:37.991652 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:37.991655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:37.992094 (+ 439us) server_negotiation.cc:407] Connection header received
0928 12:31:37.992755 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:37.992756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:37.992766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:37.992849 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:37.994573 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:37.994575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:37.994576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:37.994581 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:37.994601 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:37.994606 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:37.994671 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:37.994793 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 12:31:37.994809 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:31:37.995111 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:37.991607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:44215 (local address 127.8.13.1:41557)
0928 12:31:37.991911 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:37.991971 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:37.992618 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:37.992936 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:37.992941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:37.992953 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:37.994336 (+ 1383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:37.994337 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:37.994380 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:37.994425 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:37.994704 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:37.994711 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:37.994715 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:37.994803 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 12:31:37.994876 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20260928 12:31:37.995724 18158 heartbeater.cc:347] Connected to a master server at 127.8.13.62:44215
I20260928 12:31:37.995831 18158 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:37.995965 18158 heartbeater.cc:511] Master 127.8.13.62:44215 requested a full tablet report, sending...
I20260928 12:31:37.996622 17970 ts_manager.cc:194] Registered new tserver with Master: 880da60384f64078bd5ab8d2ae6daa25 (127.8.13.1:39269)
I20260928 12:31:37.999536 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 18030
I20260928 12:31:37.999635 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:31:37.999830 8244 external_mini_cluster.cc:1428] Setting key 41cf14529a6ff96a92a99597076b4ce4
I20260928 12:31:38.001026 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44215
--builtin_ntp_servers=127.8.13.20:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:31:38.092720 18161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:38.092960 18161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:38.092986 18161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:38.095206 18161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:38.095300 18161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:38.095340 18161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:38.097314 18161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:38.097617 18161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:38.097962 18161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:31:38.103709 18161 server_base.cc:1034] running on GCE node
W20260928 12:31:38.103665 18166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:38.103744 18169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:38.103665 18167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:38.104120 18161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:38.104419 18161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:38.105597 18161 hybrid_clock.cc:648] HybridClock initialized: now 1790598698105568 us; error 47 us; skew 500 ppm
I20260928 12:31:38.107115 18161 webserver.cc:460] Webserver started at http://127.8.13.2:36759/ using document root <none> and password file <none>
I20260928 12:31:38.107376 18161 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:38.107429 18161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:38.107560 18161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:38.108639 18161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "1c80cfea3c2b4f469a27fb69fec42a0a"
format_stamp: "Formatted at 2026-09-28 12:31:38 on dist-test-slave-g31s"
server_key: "6969ce1f45cec535a4370b80988ab5a5"
server_key_iv: "d3aeaa2fd7bb6b37b21915234f4def88"
server_key_version: "encryptionkey@0"
I20260928 12:31:38.108978 18161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "1c80cfea3c2b4f469a27fb69fec42a0a"
format_stamp: "Formatted at 2026-09-28 12:31:38 on dist-test-slave-g31s"
server_key: "6969ce1f45cec535a4370b80988ab5a5"
server_key_iv: "d3aeaa2fd7bb6b37b21915234f4def88"
server_key_version: "encryptionkey@0"
I20260928 12:31:38.110457 18161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 12:31:38.111215 18175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:38.111392 18161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:38.111471 18161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "1c80cfea3c2b4f469a27fb69fec42a0a"
format_stamp: "Formatted at 2026-09-28 12:31:38 on dist-test-slave-g31s"
server_key: "6969ce1f45cec535a4370b80988ab5a5"
server_key_iv: "d3aeaa2fd7bb6b37b21915234f4def88"
server_key_version: "encryptionkey@0"
I20260928 12:31:38.111536 18161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:38.115018 18161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:38.115268 18161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:38.115383 18161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:38.115557 18161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:38.115870 18161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:38.115903 18161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:38.115931 18161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:38.115953 18161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:38.121450 18161 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:37349
I20260928 12:31:38.121551 18288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:37349 every 8 connection(s)
I20260928 12:31:38.121863 18161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:31:38.124792 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:38.122592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:58587 (local address 127.8.13.62:44215)
0928 12:31:38.122657 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:38.122660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:38.122952 (+ 292us) server_negotiation.cc:407] Connection header received
0928 12:31:38.123403 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:38.123405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:38.123411 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:38.123468 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:38.124571 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:38.124573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:38.124573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:38.124577 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:38.124594 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:38.124598 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:38.124655 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:38.124724 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 12:31:38.124736 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:31:38.124990 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:38.122555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:44215 (local address 127.8.13.2:58587)
0928 12:31:38.122766 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:38.122821 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:38.123268 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:38.123520 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:38.123524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:38.123532 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:38.124392 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:38.124392 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:38.124422 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:38.124454 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:38.124661 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:38.124664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:38.124667 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:31:38.124722 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 12:31:38.124784 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260928 12:31:38.125443 18289 heartbeater.cc:347] Connected to a master server at 127.8.13.62:44215
I20260928 12:31:38.125531 18289 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:38.125654 18289 heartbeater.cc:511] Master 127.8.13.62:44215 requested a full tablet report, sending...
I20260928 12:31:38.126083 17970 ts_manager.cc:194] Registered new tserver with Master: 1c80cfea3c2b4f469a27fb69fec42a0a (127.8.13.2:37349)
I20260928 12:31:38.127424 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 18161
I20260928 12:31:38.127568 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:31:38.127748 8244 external_mini_cluster.cc:1428] Setting key 4343e4356fe4ef1f8e1d21aab2a09f8f
I20260928 12:31:38.128787 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44215
--builtin_ntp_servers=127.8.13.20:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:31:38.219652 18292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:38.219929 18292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:38.219957 18292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:38.221426 18292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:38.221498 18292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:38.221530 18292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:38.223152 18292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:44215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:38.223403 18292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:38.223680 18292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:38.229270 18298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:38.229378 18292 server_base.cc:1034] running on GCE node
W20260928 12:31:38.229301 18297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:38.229295 18300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:38.229959 18292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:38.230437 18292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:38.231647 18292 hybrid_clock.cc:648] HybridClock initialized: now 1790598698231610 us; error 57 us; skew 500 ppm
I20260928 12:31:38.233049 18292 webserver.cc:460] Webserver started at http://127.8.13.3:35743/ using document root <none> and password file <none>
I20260928 12:31:38.233297 18292 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:38.233349 18292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:38.233446 18292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:38.234714 18292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "f44d173776bf4cabbb6c7d306a7be45e"
format_stamp: "Formatted at 2026-09-28 12:31:38 on dist-test-slave-g31s"
server_key: "8d3b9e7ef0a0ce4199f1e3292ce233eb"
server_key_iv: "42ca122d978674a4204f2a397c4ce42f"
server_key_version: "encryptionkey@0"
I20260928 12:31:38.235040 18292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "f44d173776bf4cabbb6c7d306a7be45e"
format_stamp: "Formatted at 2026-09-28 12:31:38 on dist-test-slave-g31s"
server_key: "8d3b9e7ef0a0ce4199f1e3292ce233eb"
server_key_iv: "42ca122d978674a4204f2a397c4ce42f"
server_key_version: "encryptionkey@0"
I20260928 12:31:38.236526 18292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:31:38.237480 18306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:38.238472 18292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:38.238653 18292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "f44d173776bf4cabbb6c7d306a7be45e"
format_stamp: "Formatted at 2026-09-28 12:31:38 on dist-test-slave-g31s"
server_key: "8d3b9e7ef0a0ce4199f1e3292ce233eb"
server_key_iv: "42ca122d978674a4204f2a397c4ce42f"
server_key_version: "encryptionkey@0"
I20260928 12:31:38.238783 18292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:38.244758 18292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:38.245147 18292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:38.245410 18292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:38.245673 18292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:38.246121 18292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:38.246162 18292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:38.246199 18292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:38.246227 18292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:38.254012 18292 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:33505
I20260928 12:31:38.254117 18419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:33505 every 8 connection(s)
I20260928 12:31:38.254546 18292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:31:38.258599 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:38.255628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:35279 (local address 127.8.13.62:44215)
0928 12:31:38.255778 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:38.255780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:38.256051 (+ 271us) server_negotiation.cc:407] Connection header received
0928 12:31:38.256618 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:38.256619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:38.256628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:38.256669 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:38.258334 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:38.258336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:38.258336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:31:38.258342 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:38.258361 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:31:38.258367 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:38.258421 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:38.258496 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 12:31:38.258514 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 12:31:38.258822 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:38.255539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:44215 (local address 127.8.13.3:35279)
0928 12:31:38.255889 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:38.255942 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:38.256499 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:38.256762 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:38.256766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:38.256777 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:38.258140 (+ 1363us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:31:38.258141 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:38.258181 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:31:38.258221 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:38.258421 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:38.258427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:38.258431 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:38.258506 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:31:38.258584 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":112,"threads_started":1}
I20260928 12:31:38.259544 18420 heartbeater.cc:347] Connected to a master server at 127.8.13.62:44215
I20260928 12:31:38.259706 18420 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:38.259903 18420 heartbeater.cc:511] Master 127.8.13.62:44215 requested a full tablet report, sending...
I20260928 12:31:38.260510 17970 ts_manager.cc:194] Registered new tserver with Master: f44d173776bf4cabbb6c7d306a7be45e (127.8.13.3:33505)
I20260928 12:31:38.264695 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 18292
I20260928 12:31:38.264812 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:31:38.264969 8244 external_mini_cluster.cc:1428] Setting key a711b454da8ae46bb3dbc90306c819c1
I20260928 12:31:38.266330 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:31:38.268805 18010 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:31:38.268237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.8.13.62:44215)
0928 12:31:38.268403 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:38.268406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:38.268421 (+ 15us) server_negotiation.cc:407] Connection header received
0928 12:31:38.268508 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:38.268508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:38.268515 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:38.268562 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:38.268729 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46952: BlockingRecv error: recv got EOF from 127.0.0.1:46952 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:31:38.269079 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 18030
I20260928 12:31:38.273640 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 18161
I20260928 12:31:38.278751 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 18292
I20260928 12:31:38.283221 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 17939
2026-09-28T12:31:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (610 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:31:38 dist-test-slave-g31s krb5kdc[18430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:31:38 dist-test-slave-g31s krb5kdc[18430](info): set up 2 sockets
Sep 28 12:31:38 dist-test-slave-g31s krb5kdc[18430](info): commencing operation
krb5kdc: starting...
W20260928 12:31:39.266856 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:31:39 dist-test-slave-g31s krb5kdc[18430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:31:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:31:39.398022 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:34991
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:34991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:39.482189 18443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:39.482439 18443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:39.482476 18443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:39.483822 18443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:31:39.483891 18443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:39.483909 18443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:39.483925 18443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:31:39.483942 18443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:31:39.485623 18443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:34991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:34991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:39.485940 18443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:39.486243 18443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:39.491887 18451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:39.491887 18448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:39.492274 18449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:39.492316 18443 server_base.cc:1034] running on GCE node
I20260928 12:31:39.492597 18443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:39.493052 18443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:39.494234 18443 hybrid_clock.cc:648] HybridClock initialized: now 1790598699494197 us; error 65 us; skew 500 ppm
Sep 28 12:31:39 dist-test-slave-g31s krb5kdc[18430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598699, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:39.497437 18443 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:31:39.498023 18443 webserver.cc:460] Webserver started at http://127.8.13.62:42085/ using document root <none> and password file <none>
I20260928 12:31:39.498241 18443 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:39.498293 18443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:39.498428 18443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:39.499572 18443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "45fb96fcf8c641568d01bcfd976f432b"
format_stamp: "Formatted at 2026-09-28 12:31:39 on dist-test-slave-g31s"
server_key: "b22ede5e1bd2bde4a7b2a94eb6a8655d"
server_key_iv: "f4646be89c6b9af1e5d88dfd8cb36de1"
server_key_version: "encryptionkey@0"
I20260928 12:31:39.499933 18443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "45fb96fcf8c641568d01bcfd976f432b"
format_stamp: "Formatted at 2026-09-28 12:31:39 on dist-test-slave-g31s"
server_key: "b22ede5e1bd2bde4a7b2a94eb6a8655d"
server_key_iv: "f4646be89c6b9af1e5d88dfd8cb36de1"
server_key_version: "encryptionkey@0"
I20260928 12:31:39.501327 18443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:31:39.502246 18458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:39.502477 18443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:39.502569 18443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "45fb96fcf8c641568d01bcfd976f432b"
format_stamp: "Formatted at 2026-09-28 12:31:39 on dist-test-slave-g31s"
server_key: "b22ede5e1bd2bde4a7b2a94eb6a8655d"
server_key_iv: "f4646be89c6b9af1e5d88dfd8cb36de1"
server_key_version: "encryptionkey@0"
I20260928 12:31:39.502641 18443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:39.517383 18443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:39.519290 18443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:39.519421 18443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:39.523100 18443 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:34991
I20260928 12:31:39.523155 18510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:34991 every 8 connection(s)
I20260928 12:31:39.523463 18443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:31:39.524371 18511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:39.526542 18511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b: Bootstrap starting.
I20260928 12:31:39.527041 18511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:39.527258 18511 log.cc:824] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:39.527892 18511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b: No bootstrap required, opened a new log
I20260928 12:31:39.529436 18511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fb96fcf8c641568d01bcfd976f432b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34991 } }
I20260928 12:31:39.529569 18511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:39.529589 18511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fb96fcf8c641568d01bcfd976f432b, State: Initialized, Role: FOLLOWER
I20260928 12:31:39.529692 18511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fb96fcf8c641568d01bcfd976f432b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34991 } }
I20260928 12:31:39.529760 18511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:31:39.529783 18511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:31:39.529861 18511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:39.530637 18511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fb96fcf8c641568d01bcfd976f432b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34991 } }
I20260928 12:31:39.530727 18511 leader_election.cc:302] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45fb96fcf8c641568d01bcfd976f432b; no voters:
I20260928 12:31:39.530877 18511 leader_election.cc:288] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:31:39.530933 18514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:39.531085 18514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [term 1 LEADER]: Becoming Leader. State: Replica: 45fb96fcf8c641568d01bcfd976f432b, State: Running, Role: LEADER
I20260928 12:31:39.531112 18511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:31:39.531180 18514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fb96fcf8c641568d01bcfd976f432b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34991 } }
I20260928 12:31:39.531538 18515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45fb96fcf8c641568d01bcfd976f432b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fb96fcf8c641568d01bcfd976f432b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34991 } } }
I20260928 12:31:39.531564 18516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45fb96fcf8c641568d01bcfd976f432b. Latest consensus state: current_term: 1 leader_uuid: "45fb96fcf8c641568d01bcfd976f432b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fb96fcf8c641568d01bcfd976f432b" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34991 } } }
I20260928 12:31:39.531628 18516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:39.531625 18515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b [sys.catalog]: This master's current role is: LEADER
I20260928 12:31:39.531842 18523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:31:39.532349 18523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:31:39.533618 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 18443
I20260928 12:31:39.533722 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:31:39.533855 18523 catalog_manager.cc:1347] Generated new cluster ID: 75688f7faef6486aabade5971e6871d4
I20260928 12:31:39.533907 18523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:31:39.533949 8244 external_mini_cluster.cc:1428] Setting key 9804f47431f897ce8d9883649c824f77
Sep 28 12:31:39 dist-test-slave-g31s krb5kdc[18430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:39.545940 18523 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:31:39.546866 18523 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:31:39.549423 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:39.534380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.8.13.62:34991)
0928 12:31:39.538209 (+ 3829us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:39.538220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:39.538256 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:31:39.538405 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:39.538412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:39.538462 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:39.538594 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:39.539260 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:39.540542 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:39.541276 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:39.541711 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:39.541836 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:39.544225 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:39.544234 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:39.544234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:39.544253 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:39.547358 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:39.547893 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:39.547897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:39.547898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:39.548039 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:39.548247 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:39.548248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:39.548248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:39.548571 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:39.548730 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:39.549007 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 12:31:39.549185 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3775,"thread_start_us":63,"threads_started":1}
I20260928 12:31:39.549839 18523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b: Generated new TSK 0
I20260928 12:31:39.549997 18523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:31:39.585856 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34991
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:39.669524 18537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:39.669770 18537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:39.669827 18537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:39.671248 18537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:39.671306 18537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:39.671348 18537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:31:39.673008 18537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:39.673343 18537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:39.673616 18537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:39.679029 18545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:39.679108 18537 server_base.cc:1034] running on GCE node
W20260928 12:31:39.679029 18543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:39.679059 18542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:39.679548 18537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:39.679813 18537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:39.680984 18537 hybrid_clock.cc:648] HybridClock initialized: now 1790598699680967 us; error 29 us; skew 500 ppm
Sep 28 12:31:39 dist-test-slave-g31s krb5kdc[18430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598699, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:39.683862 18537 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:31:39.684535 18537 webserver.cc:460] Webserver started at http://127.8.13.1:34293/ using document root <none> and password file <none>
I20260928 12:31:39.684787 18537 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:39.684840 18537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:39.684971 18537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:39.686486 18537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "ee9238937bba42cebc6087a3e5d5f4eb"
format_stamp: "Formatted at 2026-09-28 12:31:39 on dist-test-slave-g31s"
server_key: "de31b413d17f9129c8ec70285ae3b531"
server_key_iv: "185e7a841f879d962f5c7654aa93098d"
server_key_version: "encryptionkey@0"
I20260928 12:31:39.686844 18537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "ee9238937bba42cebc6087a3e5d5f4eb"
format_stamp: "Formatted at 2026-09-28 12:31:39 on dist-test-slave-g31s"
server_key: "de31b413d17f9129c8ec70285ae3b531"
server_key_iv: "185e7a841f879d962f5c7654aa93098d"
server_key_version: "encryptionkey@0"
I20260928 12:31:39.688529 18537 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 12:31:39.689376 18552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:39.689738 18537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:31:39.689963 18537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "ee9238937bba42cebc6087a3e5d5f4eb"
format_stamp: "Formatted at 2026-09-28 12:31:39 on dist-test-slave-g31s"
server_key: "de31b413d17f9129c8ec70285ae3b531"
server_key_iv: "185e7a841f879d962f5c7654aa93098d"
server_key_version: "encryptionkey@0"
I20260928 12:31:39.690053 18537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:39.708745 18537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:39.710826 18537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:39.710963 18537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:39.711218 18537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:39.711534 18537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:39.711565 18537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:39.711596 18537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:39.711613 18537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:39.717010 18537 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:39425
I20260928 12:31:39.717069 18665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:39425 every 8 connection(s)
I20260928 12:31:39.717361 18537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:31:39.721415 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 18537
I20260928 12:31:39.721516 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:31:39.721676 8244 external_mini_cluster.cc:1428] Setting key f41b9e39fb55bb03e2c65a0270c99f1b
Sep 28 12:31:39 dist-test-slave-g31s krb5kdc[18430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598699, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:39.726889 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:39.718094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:42053 (local address 127.8.13.62:34991)
0928 12:31:39.718186 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:39.718188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:39.718357 (+ 169us) server_negotiation.cc:407] Connection header received
0928 12:31:39.718882 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:39.718884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:39.718899 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:39.718949 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:39.719895 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:39.720405 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:39.721392 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:39.721612 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:39.721670 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:39.723956 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:39.723958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:39.723959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:39.723966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:39.725371 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:39.725892 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:39.725894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:39.725894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:39.725957 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:39.726180 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:39.726181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:39.726181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:39.726323 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:39.726396 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:39.726688 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 12:31:39.726785 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 12:31:39.726970 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:39.718034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34991 (local address 127.8.13.1:42053)
0928 12:31:39.718235 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:39.718275 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:39.718780 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:39.719016 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:39.719019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:39.719601 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:39.719796 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:39.719801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:39.720489 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:39.720491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:39.721308 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:39.721310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:39.721678 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:39.721679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:39.721852 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:39.722385 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:39.722393 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:39.723861 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:39.725475 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:39.725483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:39.725484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:39.725769 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:39.726019 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:39.726020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:39.726020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:39.726125 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:39.726412 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:39.726418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:39.726547 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 12:31:39.726645 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 12:31:39.726789 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":90,"threads_started":1}
I20260928 12:31:39.727541 18666 heartbeater.cc:347] Connected to a master server at 127.8.13.62:34991
I20260928 12:31:39.727634 18666 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:39.727811 18666 heartbeater.cc:511] Master 127.8.13.62:34991 requested a full tablet report, sending...
I20260928 12:31:39.728430 18475 ts_manager.cc:194] Registered new tserver with Master: ee9238937bba42cebc6087a3e5d5f4eb (127.8.13.1:39425)
I20260928 12:31:39.729041 18475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:42053
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:31:39.750195 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34991
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:39.834684 18671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:39.834934 18671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:39.834961 18671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:39.836277 18671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:39.836321 18671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:39.836350 18671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:31:39.837905 18671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:39.838120 18671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:39.838325 18671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:39.843904 18679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:39.843952 18676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:39.844164 18677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:39.844211 18671 server_base.cc:1034] running on GCE node
I20260928 12:31:39.844534 18671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:39.844902 18671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:39.846103 18671 hybrid_clock.cc:648] HybridClock initialized: now 1790598699846083 us; error 44 us; skew 500 ppm
Sep 28 12:31:39 dist-test-slave-g31s krb5kdc[18430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598699, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:39.848956 18671 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:31:39.849439 18671 webserver.cc:460] Webserver started at http://127.8.13.2:42981/ using document root <none> and password file <none>
I20260928 12:31:39.849644 18671 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:39.849689 18671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:39.849840 18671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:39.851528 18671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "c6075cc497bd4b37b5c9c4d03e6e6424"
format_stamp: "Formatted at 2026-09-28 12:31:39 on dist-test-slave-g31s"
server_key: "9e47bf123ec474ce2617d49e0740ab53"
server_key_iv: "ebd5ef2c4f04548931fcb9ef9cd8a85d"
server_key_version: "encryptionkey@0"
I20260928 12:31:39.851962 18671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "c6075cc497bd4b37b5c9c4d03e6e6424"
format_stamp: "Formatted at 2026-09-28 12:31:39 on dist-test-slave-g31s"
server_key: "9e47bf123ec474ce2617d49e0740ab53"
server_key_iv: "ebd5ef2c4f04548931fcb9ef9cd8a85d"
server_key_version: "encryptionkey@0"
I20260928 12:31:39.853367 18671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:31:39.854249 18686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:39.854462 18671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:31:39.854555 18671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "c6075cc497bd4b37b5c9c4d03e6e6424"
format_stamp: "Formatted at 2026-09-28 12:31:39 on dist-test-slave-g31s"
server_key: "9e47bf123ec474ce2617d49e0740ab53"
server_key_iv: "ebd5ef2c4f04548931fcb9ef9cd8a85d"
server_key_version: "encryptionkey@0"
I20260928 12:31:39.854622 18671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:39.866691 18671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:39.869379 18671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:39.869580 18671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:39.870031 18671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:39.870584 18671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:39.870668 18671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:39.870708 18671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:39.870724 18671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:39.876907 18671 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:34923
I20260928 12:31:39.877015 18799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:34923 every 8 connection(s)
I20260928 12:31:39.877344 18671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:31:39 dist-test-slave-g31s krb5kdc[18430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598699, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:39.885521 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 18671
I20260928 12:31:39.885634 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:31:39.885927 8244 external_mini_cluster.cc:1428] Setting key b46d953814ee5ee40c3dfeb42d6a8179
I20260928 12:31:39.889276 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:39.878278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:36679 (local address 127.8.13.62:34991)
0928 12:31:39.878347 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:39.878350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:39.878527 (+ 177us) server_negotiation.cc:407] Connection header received
0928 12:31:39.879049 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:39.879050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:39.879060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:39.879109 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:39.880165 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:39.880777 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:39.882092 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:39.882319 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:39.882393 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:39.885195 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:39.885198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:39.885199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:39.885208 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:39.887119 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:39.887745 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:39.887747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:39.887747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:39.887815 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:39.888334 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:39.888335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:39.888335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:39.888502 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:39.888586 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:39.889082 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 12:31:39.889180 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:31:39.889667 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:39.878230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34991 (local address 127.8.13.2:36679)
0928 12:31:39.878405 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:39.878445 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:39.878946 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:39.879174 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:39.879178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:39.879782 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:39.879990 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:39.879995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:39.880957 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:39.880959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:39.881927 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:39.881929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:39.882438 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:39.882440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:39.882566 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:39.883216 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:39.883224 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:39.884999 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:39.887274 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:39.887287 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:39.887291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:39.887628 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:39.888104 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:39.888104 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:39.888105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:39.888264 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:39.888641 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:39.888657 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:39.888897 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 12:31:39.889035 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 12:31:39.889246 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260928 12:31:39.890430 18800 heartbeater.cc:347] Connected to a master server at 127.8.13.62:34991
I20260928 12:31:39.890545 18800 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:39.890744 18800 heartbeater.cc:511] Master 127.8.13.62:34991 requested a full tablet report, sending...
I20260928 12:31:39.891258 18475 ts_manager.cc:194] Registered new tserver with Master: c6075cc497bd4b37b5c9c4d03e6e6424 (127.8.13.2:34923)
I20260928 12:31:39.891692 18475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:36679
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:31:39.914382 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34991
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:31:39.999991 18805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:31:40.000232 18805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:31:40.000262 18805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:31:40.001642 18805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:31:40.001705 18805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:31:40.001737 18805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:31:40.003491 18805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:31:40.003722 18805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:31:40.003933 18805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:31:40.009332 18813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:40.009346 18811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:31:40.009339 18810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:31:40.009650 18805 server_base.cc:1034] running on GCE node
I20260928 12:31:40.009912 18805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:31:40.010164 18805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:31:40.011317 18805 hybrid_clock.cc:648] HybridClock initialized: now 1790598700011302 us; error 29 us; skew 500 ppm
Sep 28 12:31:40 dist-test-slave-g31s krb5kdc[18430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598700, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:31:40.013979 18805 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:31:40.014396 18805 webserver.cc:460] Webserver started at http://127.8.13.3:32821/ using document root <none> and password file <none>
I20260928 12:31:40.014624 18805 fs_manager.cc:359] Metadata directory not provided
I20260928 12:31:40.014673 18805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:31:40.014779 18805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:31:40.015828 18805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "bcf07c9dadf84eb89d02567016d50ae7"
format_stamp: "Formatted at 2026-09-28 12:31:40 on dist-test-slave-g31s"
server_key: "a8594b4c015ec84e09640ec014bed026"
server_key_iv: "b03ffa33c6761373d0389e23affacf07"
server_key_version: "encryptionkey@0"
I20260928 12:31:40.016120 18805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "bcf07c9dadf84eb89d02567016d50ae7"
format_stamp: "Formatted at 2026-09-28 12:31:40 on dist-test-slave-g31s"
server_key: "a8594b4c015ec84e09640ec014bed026"
server_key_iv: "b03ffa33c6761373d0389e23affacf07"
server_key_version: "encryptionkey@0"
I20260928 12:31:40.017400 18805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:31:40.018121 18820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:40.018268 18805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:40.018334 18805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "bcf07c9dadf84eb89d02567016d50ae7"
format_stamp: "Formatted at 2026-09-28 12:31:40 on dist-test-slave-g31s"
server_key: "a8594b4c015ec84e09640ec014bed026"
server_key_iv: "b03ffa33c6761373d0389e23affacf07"
server_key_version: "encryptionkey@0"
I20260928 12:31:40.018405 18805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:31:40.028874 18805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:31:40.031050 18805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:31:40.031190 18805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:31:40.031441 18805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:31:40.031785 18805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:31:40.031819 18805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:40.031849 18805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:31:40.031867 18805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:31:40.037532 18805 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:42685
I20260928 12:31:40.037592 18933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:42685 every 8 connection(s)
I20260928 12:31:40.037988 18805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:31:40.039179 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 18805
I20260928 12:31:40.039301 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:31:40.039480 8244 external_mini_cluster.cc:1428] Setting key 827361662b74e264234e24ea3e94fa0c
Sep 28 12:31:40 dist-test-slave-g31s krb5kdc[18430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598700, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:31:40.047962 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.038762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:50285 (local address 127.8.13.62:34991)
0928 12:31:40.038820 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:40.038822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:40.038989 (+ 167us) server_negotiation.cc:407] Connection header received
0928 12:31:40.039573 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:40.039574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:40.039587 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:40.039636 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:40.040855 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.041369 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.042476 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.042710 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.042768 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.044941 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:40.044943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:40.044943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:40.044952 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:40.046542 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:40.047002 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:40.047003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:40.047003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:40.047057 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:40.047291 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:40.047292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:40.047292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:40.047420 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:40.047476 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:40.047769 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 12:31:40.047865 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:31:40.048148 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.038717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34991 (local address 127.8.13.3:50285)
0928 12:31:40.038866 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:40.038911 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:40.039453 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:40.039897 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:40.039902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:40.040537 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:31:40.040759 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.040764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.041458 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.041459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.042382 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.042385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.042788 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.042789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.042919 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.043421 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:31:40.043430 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:31:40.044839 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:31:40.046674 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:40.046684 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:40.046687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:40.046928 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:40.047112 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:31:40.047112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:31:40.047113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:31:40.047234 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:31:40.047478 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:31:40.047484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:31:40.047635 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 12:31:40.047737 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 12:31:40.047912 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260928 12:31:40.048653 18934 heartbeater.cc:347] Connected to a master server at 127.8.13.62:34991
I20260928 12:31:40.048745 18934 heartbeater.cc:464] Registering TS with master...
I20260928 12:31:40.048926 18934 heartbeater.cc:511] Master 127.8.13.62:34991 requested a full tablet report, sending...
I20260928 12:31:40.049376 18475 ts_manager.cc:194] Registered new tserver with Master: bcf07c9dadf84eb89d02567016d50ae7 (127.8.13.3:42685)
I20260928 12:31:40.050040 18475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:50285
I20260928 12:31:40.051311 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:31:40.059504 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.052976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.8.13.62:34991)
0928 12:31:40.053050 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:40.053052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:40.053130 (+ 78us) server_negotiation.cc:407] Connection header received
0928 12:31:40.053212 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:40.053213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:40.053222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:40.053284 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:31:40.053884 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.054398 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.055167 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.055365 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.055428 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.056734 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:31:40.056736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:31:40.056736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:31:40.056743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:31:40.058270 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:40.058683 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:40.058684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:40.058685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:40.058749 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:31:40.058961 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:31:40.058962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:31:40.058962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:31:40.059134 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:31:40.059215 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:40.059352 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 12:31:40.059430 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:31:40.061247 18475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:31:40.063021 18475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:31:40.070634 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.066342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.8.13.1:39425)
0928 12:31:40.066654 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:40.066658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:40.066668 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:31:40.066709 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:40.066711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:40.066727 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:40.066801 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:40.066991 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.068169 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.069554 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.070196 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.070272 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.070338 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:40.070490 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 12:31:40.070532 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":70,"threads_started":1}
I20260928 12:31:40.070621 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.066159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39425 (local address 127.0.0.1:43126)
0928 12:31:40.066350 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:40.066392 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:40.066462 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:40.066819 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:40.066822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:40.066825 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:40.066907 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.066909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.068313 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.068314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.069351 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.069359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.070285 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.070287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.070383 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.070399 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 12:31:40.070466 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 12:31:40.070532 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260928 12:31:40.070978 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.066159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:34923 (local address 127.0.0.1:49242)
0928 12:31:40.066350 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:40.066392 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:40.066462 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:40.066884 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:40.066885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:40.066886 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:40.066916 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.066917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.068276 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.068277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.069361 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.069364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.070589 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.070591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.070657 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.070661 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:40.070691 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 12:31:40.070737 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":80,"threads_started":1}
I20260928 12:31:40.071461 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.066523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.8.13.2:34923)
0928 12:31:40.066690 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:40.066695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:40.066707 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:31:40.066754 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:40.066756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:40.066769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:40.066981 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:40.067046 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.068189 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.069486 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.070468 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.071198 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.071283 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:40.071317 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 12:31:40.071374 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260928 12:31:40.071411 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb87837cb2f49309d1323200b9ca4b3 (DEFAULT_TABLE table=test-table [id=d665a796e77f4bd1a0a8e8776941e87f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:40.071704 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.066221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:42685 (local address 127.0.0.1:56926)
0928 12:31:40.066903 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:40.066911 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:40.066956 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:40.067138 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:40.067138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:40.067140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:40.067195 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.067198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.068050 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.068051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.069580 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.069583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.071382 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.071384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.071452 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.071458 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:31:40.071591 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 12:31:40.071642 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":109,"threads_started":1}
I20260928 12:31:40.071844 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb87837cb2f49309d1323200b9ca4b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:40.072016 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.066473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.8.13.3:42685)
0928 12:31:40.066648 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:40.066653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:40.067006 (+ 353us) server_negotiation.cc:407] Connection header received
0928 12:31:40.067047 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:40.067050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:40.067064 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:40.067127 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:40.067262 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.067949 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.069669 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.070636 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.071366 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.071440 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:40.071861 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 12:31:40.071930 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260928 12:31:40.072607 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb87837cb2f49309d1323200b9ca4b3 (DEFAULT_TABLE table=test-table [id=d665a796e77f4bd1a0a8e8776941e87f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:40.072718 18868 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb87837cb2f49309d1323200b9ca4b3 (DEFAULT_TABLE table=test-table [id=d665a796e77f4bd1a0a8e8776941e87f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:31:40.073035 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb87837cb2f49309d1323200b9ca4b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:40.073113 18868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb87837cb2f49309d1323200b9ca4b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:31:40.075521 18951 tablet_bootstrap.cc:492] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb: Bootstrap starting.
I20260928 12:31:40.076452 18951 tablet_bootstrap.cc:654] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:40.076470 18952 tablet_bootstrap.cc:492] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: Bootstrap starting.
I20260928 12:31:40.077099 18952 tablet_bootstrap.cc:654] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:40.077222 18951 log.cc:824] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:40.077663 18952 log.cc:824] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:40.077709 18953 tablet_bootstrap.cc:492] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: Bootstrap starting.
I20260928 12:31:40.078446 18953 tablet_bootstrap.cc:654] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: Neither blocks nor log segments found. Creating new log.
I20260928 12:31:40.078580 18951 tablet_bootstrap.cc:492] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb: No bootstrap required, opened a new log
I20260928 12:31:40.078580 18952 tablet_bootstrap.cc:492] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: No bootstrap required, opened a new log
I20260928 12:31:40.078679 18951 ts_tablet_manager.cc:1397] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:31:40.078681 18952 ts_tablet_manager.cc:1397] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:31:40.078979 18953 log.cc:824] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: Log is configured to *not* fsync() on all Append() calls
I20260928 12:31:40.079720 18953 tablet_bootstrap.cc:492] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: No bootstrap required, opened a new log
I20260928 12:31:40.079788 18953 ts_tablet_manager.cc:1397] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:31:40.080789 18951 raft_consensus.cc:348] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.080788 18952 raft_consensus.cc:348] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.081009 18951 raft_consensus.cc:374] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:40.081009 18952 raft_consensus.cc:374] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:40.081045 18952 raft_consensus.cc:729] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf07c9dadf84eb89d02567016d50ae7, State: Initialized, Role: FOLLOWER
I20260928 12:31:40.081045 18951 raft_consensus.cc:729] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9238937bba42cebc6087a3e5d5f4eb, State: Initialized, Role: FOLLOWER
I20260928 12:31:40.081171 18951 consensus_queue.cc:260] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.081171 18952 consensus_queue.cc:260] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.081498 18952 ts_tablet_manager.cc:1428] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:31:40.081498 18951 ts_tablet_manager.cc:1428] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:31:40.081741 18934 heartbeater.cc:503] Master 127.8.13.62:34991 was elected leader, sending a full tablet report...
I20260928 12:31:40.081789 18666 heartbeater.cc:503] Master 127.8.13.62:34991 was elected leader, sending a full tablet report...
I20260928 12:31:40.082433 18953 raft_consensus.cc:348] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.082597 18953 raft_consensus.cc:374] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:31:40.082638 18953 raft_consensus.cc:729] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6075cc497bd4b37b5c9c4d03e6e6424, State: Initialized, Role: FOLLOWER
I20260928 12:31:40.082773 18953 consensus_queue.cc:260] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.083125 18800 heartbeater.cc:503] Master 127.8.13.62:34991 was elected leader, sending a full tablet report...
I20260928 12:31:40.083127 18953 ts_tablet_manager.cc:1428] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20260928 12:31:40.128012 18801 tablet.cc:2369] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:31:40.218047 18667 tablet.cc:2369] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:31:40.288605 18935 tablet.cc:2369] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:31:40.346172 18957 raft_consensus.cc:482] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:40.346379 18957 raft_consensus.cc:504] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.346882 18957 leader_election.cc:288] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6075cc497bd4b37b5c9c4d03e6e6424 (127.8.13.2:34923), bcf07c9dadf84eb89d02567016d50ae7 (127.8.13.3:42685)
I20260928 12:31:40.348600 18959 raft_consensus.cc:482] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:31:40.348749 18959 raft_consensus.cc:504] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.349202 18959 leader_election.cc:288] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee9238937bba42cebc6087a3e5d5f4eb (127.8.13.1:39425), bcf07c9dadf84eb89d02567016d50ae7 (127.8.13.3:42685)
I20260928 12:31:40.352355 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.347287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:40173 (local address 127.8.13.3:42685)
0928 12:31:40.347352 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:40.347354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:40.347531 (+ 177us) server_negotiation.cc:407] Connection header received
0928 12:31:40.347579 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:40.347580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:40.347597 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:40.347671 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:40.348179 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.348976 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.350853 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.351499 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.351592 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.351610 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:40.352169 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 12:31:40.352236 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:31:40.352388 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.347292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:40163 (local address 127.8.13.2:34923)
0928 12:31:40.347428 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:40.347431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:40.347531 (+ 100us) server_negotiation.cc:407] Connection header received
0928 12:31:40.347579 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:40.347580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:40.347598 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:40.347671 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:40.348173 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.348977 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.350805 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.351389 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.351442 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.351456 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:40.352248 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 12:31:40.352305 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 12:31:40.352880 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.349349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39425 (local address 127.8.13.2:55867)
0928 12:31:40.349410 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:40.349422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:40.349459 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:40.349674 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:40.349676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:40.349678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:40.349741 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.349742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.350641 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.350642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.351876 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.351880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.352648 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.352649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.352728 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.352732 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:40.352776 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:31:40.352817 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 12:31:40.352878 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.347173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:42685 (local address 127.8.13.1:40173)
0928 12:31:40.347387 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:40.347402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:40.347499 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:40.347789 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:40.347792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:40.347799 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:40.347973 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.347976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.349127 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.349128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.350737 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.350744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.351799 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.351800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.351900 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.351905 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:31:40.352682 (+ 777us) client_negotiation.cc:240] Negotiation successful
0928 12:31:40.352758 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":46,"thread_start_us":134,"threads_started":1}
I20260928 12:31:40.353142 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.349468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:55867 (local address 127.8.13.1:39425)
0928 12:31:40.349512 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:40.349514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:40.349521 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:31:40.349542 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:40.349543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:40.349552 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:40.349612 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:40.349840 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.350531 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.351978 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.352528 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.352596 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.352607 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:40.353052 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 12:31:40.353088 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:31:40.353262 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.347173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:34923 (local address 127.8.13.1:40163)
0928 12:31:40.347395 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:40.347404 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:40.347499 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:40.347789 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:40.347792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:40.347799 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:40.347973 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.347976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.349120 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.349146 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.350708 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.350711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.351619 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.351620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.351709 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.351714 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:31:40.352855 (+ 1141us) client_negotiation.cc:240] Negotiation successful
0928 12:31:40.352906 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":72,"thread_start_us":117,"threads_started":1}
I20260928 12:31:40.353375 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb87837cb2f49309d1323200b9ca4b3" candidate_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf07c9dadf84eb89d02567016d50ae7" is_pre_election: true
I20260928 12:31:40.353571 18888 raft_consensus.cc:2454] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9238937bba42cebc6087a3e5d5f4eb in term 0.
I20260928 12:31:40.353881 18554 leader_election.cc:302] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcf07c9dadf84eb89d02567016d50ae7, ee9238937bba42cebc6087a3e5d5f4eb; no voters:
I20260928 12:31:40.353991 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb87837cb2f49309d1323200b9ca4b3" candidate_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" is_pre_election: true
I20260928 12:31:40.354066 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.349349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:42685 (local address 127.8.13.2:33063)
0928 12:31:40.349880 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:40.349890 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:31:40.349937 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:31:40.350361 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:31:40.350363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:31:40.350367 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:31:40.350485 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.350487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.351537 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.351538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.352892 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:31:40.352895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.353794 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.353819 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:31:40.353898 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.353902 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:31:40.353935 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 12:31:40.353983 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":149,"thread_start_us":125,"threads_started":1}
I20260928 12:31:40.354044 18957 raft_consensus.cc:2781] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:31:40.354094 18957 raft_consensus.cc:482] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:31:40.354115 18957 raft_consensus.cc:3037] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:40.354300 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb87837cb2f49309d1323200b9ca4b3" candidate_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" is_pre_election: true
I20260928 12:31:40.354390 18754 raft_consensus.cc:2454] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9238937bba42cebc6087a3e5d5f4eb in term 0.
I20260928 12:31:40.354399 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.349715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:33063 (local address 127.8.13.3:42685)
0928 12:31:40.350032 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:40.350034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:40.350130 (+ 96us) server_negotiation.cc:407] Connection header received
0928 12:31:40.350180 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:40.350181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:40.350196 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:40.350549 (+ 353us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:31:40.350612 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.351422 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.353027 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.353693 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.354236 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.354254 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:40.354275 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 12:31:40.354323 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":103,"threads_started":1}
I20260928 12:31:40.354667 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb87837cb2f49309d1323200b9ca4b3" candidate_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf07c9dadf84eb89d02567016d50ae7" is_pre_election: true
I20260928 12:31:40.354761 18888 raft_consensus.cc:2454] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6075cc497bd4b37b5c9c4d03e6e6424 in term 0.
I20260928 12:31:40.354915 18688 leader_election.cc:302] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcf07c9dadf84eb89d02567016d50ae7, c6075cc497bd4b37b5c9c4d03e6e6424; no voters:
I20260928 12:31:40.355083 18957 raft_consensus.cc:504] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.355260 18957 leader_election.cc:288] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [CANDIDATE]: Term 1 election: Requested vote from peers c6075cc497bd4b37b5c9c4d03e6e6424 (127.8.13.2:34923), bcf07c9dadf84eb89d02567016d50ae7 (127.8.13.3:42685)
I20260928 12:31:40.355264 18620 raft_consensus.cc:2379] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6075cc497bd4b37b5c9c4d03e6e6424 in current term 1: Already voted for candidate ee9238937bba42cebc6087a3e5d5f4eb in this term.
I20260928 12:31:40.355413 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb87837cb2f49309d1323200b9ca4b3" candidate_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf07c9dadf84eb89d02567016d50ae7"
I20260928 12:31:40.355481 18888 raft_consensus.cc:3037] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:40.355857 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb87837cb2f49309d1323200b9ca4b3" candidate_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424"
I20260928 12:31:40.355279 18959 raft_consensus.cc:2781] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:31:40.356196 18754 raft_consensus.cc:3037] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:31:40.356208 18888 raft_consensus.cc:2454] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9238937bba42cebc6087a3e5d5f4eb in term 1.
I20260928 12:31:40.356424 18554 leader_election.cc:302] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcf07c9dadf84eb89d02567016d50ae7, ee9238937bba42cebc6087a3e5d5f4eb; no voters:
I20260928 12:31:40.356520 18957 raft_consensus.cc:2781] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:31:40.356676 18957 raft_consensus.cc:686] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 1 LEADER]: Becoming Leader. State: Replica: ee9238937bba42cebc6087a3e5d5f4eb, State: Running, Role: LEADER
I20260928 12:31:40.356781 18957 consensus_queue.cc:237] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.357286 18754 raft_consensus.cc:2454] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9238937bba42cebc6087a3e5d5f4eb in term 1.
I20260928 12:31:40.357479 18959 raft_consensus.cc:482] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:31:40.357542 18959 raft_consensus.cc:3037] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:31:40.358000 18474 catalog_manager.cc:5495] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb reported cstate change: term changed from 0 to 1, leader changed from <none> to ee9238937bba42cebc6087a3e5d5f4eb (127.8.13.1). New cstate: current_term: 1 leader_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260928 12:31:40.358536 18959 raft_consensus.cc:504] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.358868 18959 leader_election.cc:288] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [CANDIDATE]: Term 2 election: Requested vote from peers ee9238937bba42cebc6087a3e5d5f4eb (127.8.13.1:39425), bcf07c9dadf84eb89d02567016d50ae7 (127.8.13.3:42685)
I20260928 12:31:40.359256 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb87837cb2f49309d1323200b9ca4b3" candidate_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9238937bba42cebc6087a3e5d5f4eb"
I20260928 12:31:40.359341 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb87837cb2f49309d1323200b9ca4b3" candidate_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf07c9dadf84eb89d02567016d50ae7"
I20260928 12:31:40.359408 18888 raft_consensus.cc:3037] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:31:40.359987 18888 raft_consensus.cc:2454] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6075cc497bd4b37b5c9c4d03e6e6424 in term 2.
I20260928 12:31:40.360160 18688 leader_election.cc:302] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bcf07c9dadf84eb89d02567016d50ae7, c6075cc497bd4b37b5c9c4d03e6e6424; no voters: ee9238937bba42cebc6087a3e5d5f4eb
I20260928 12:31:40.360321 18959 raft_consensus.cc:2781] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:31:40.360443 18959 raft_consensus.cc:686] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 2 LEADER]: Becoming Leader. State: Replica: c6075cc497bd4b37b5c9c4d03e6e6424, State: Running, Role: LEADER
I20260928 12:31:40.360550 18959 consensus_queue.cc:237] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.361661 18474 catalog_manager.cc:5495] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 reported cstate change: term changed from 1 to 2, leader changed from ee9238937bba42cebc6087a3e5d5f4eb (127.8.13.1) to c6075cc497bd4b37b5c9c4d03e6e6424 (127.8.13.2). New cstate: current_term: 2 leader_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260928 12:31:40.426460 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.424078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.8.13.2:34923)
0928 12:31:40.424164 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 12:31:40.424166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:31:40.424174 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:31:40.424211 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:31:40.424212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:31:40.424229 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:31:40.424289 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:31:40.424439 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.424925 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.425672 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:31:40.425900 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:31:40.425954 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:31:40.426116 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:31:40.426230 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:31:40.426304 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:31:40.426336 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 12:31:40.426358 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 12:31:40.428234 18620 raft_consensus.cc:3032] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 1 LEADER]: Stepping down as leader of term 1
I20260928 12:31:40.428298 18620 raft_consensus.cc:729] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee9238937bba42cebc6087a3e5d5f4eb, State: Running, Role: LEADER
I20260928 12:31:40.428323 18888 raft_consensus.cc:1261] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 2 FOLLOWER]: Refusing update from remote peer c6075cc497bd4b37b5c9c4d03e6e6424: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260928 12:31:40.428350 18620 consensus_queue.cc:260] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 } } peers { permanent_uuid: "c6075cc497bd4b37b5c9c4d03e6e6424" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34923 } } peers { permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 } }
I20260928 12:31:40.428458 18620 raft_consensus.cc:3037] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:31:40.428643 18959 consensus_queue.cc:1035] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf07c9dadf84eb89d02567016d50ae7" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:40.429327 18620 raft_consensus.cc:1261] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 2 FOLLOWER]: Refusing update from remote peer c6075cc497bd4b37b5c9c4d03e6e6424: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260928 12:31:40.429534 18959 consensus_queue.cc:1035] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9238937bba42cebc6087a3e5d5f4eb" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:31:40.429625 18959 consensus_queue.cc:1225] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [LEADER]: Peer ee9238937bba42cebc6087a3e5d5f4eb log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260928 12:31:40.429968 18620 pending_rounds.cc:76] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb: Aborting all ops after (but not including) 0
I20260928 12:31:40.430016 18620 pending_rounds.cc:97] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260928 12:31:40.430081 18620 raft_consensus.cc:2866] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260928 12:31:40.431255 18970 mvcc.cc:205] Tried to move back new op lower bound from 7334292276951728128 to 7334292276677595136. Current Snapshot: MvccSnapshot[applied={T|T < 7334292276951728128}]
I20260928 12:31:40.432291 18972 mvcc.cc:205] Tried to move back new op lower bound from 7334292276951728128 to 7334292276677595136. Current Snapshot: MvccSnapshot[applied={T|T < 7334292276951728128}]
I20260928 12:31:40.433933 18474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:40.434049 18474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:31:40.434968 18474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45fb96fcf8c641568d01bcfd976f432b: Sending DeleteTablet for 3 replicas of tablet 4cb87837cb2f49309d1323200b9ca4b3
I20260928 12:31:40.435349 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb87837cb2f49309d1323200b9ca4b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:40 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:49242
I20260928 12:31:40.435385 18868 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb87837cb2f49309d1323200b9ca4b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:40 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:56926
I20260928 12:31:40.435418 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb87837cb2f49309d1323200b9ca4b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:31:40 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:43126
I20260928 12:31:40.435648 18978 tablet_replica.cc:330] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: stopping tablet replica
I20260928 12:31:40.435648 18980 tablet_replica.cc:330] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb: stopping tablet replica
I20260928 12:31:40.435745 18978 raft_consensus.cc:2229] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 2 LEADER]: Raft consensus shutting down.
I20260928 12:31:40.435752 18980 raft_consensus.cc:2229] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:40.435850 18980 raft_consensus.cc:2258] T 4cb87837cb2f49309d1323200b9ca4b3 P ee9238937bba42cebc6087a3e5d5f4eb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:40.435858 18978 raft_consensus.cc:2258] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:40.436021 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 18537
I20260928 12:31:40.436329 18978 ts_tablet_manager.cc:1905] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:40.436756 18979 tablet_replica.cc:330] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: stopping tablet replica
I20260928 12:31:40.436875 18979 raft_consensus.cc:2229] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260928 12:31:40.436985 18979 raft_consensus.cc:2258] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260928 12:31:40.437454 18979 ts_tablet_manager.cc:1905] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:31:40.438261 18978 ts_tablet_manager.cc:1918] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260928 12:31:40.438328 18978 log.cc:1196] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/4cb87837cb2f49309d1323200b9ca4b3
I20260928 12:31:40.438560 18978 ts_tablet_manager.cc:1939] T 4cb87837cb2f49309d1323200b9ca4b3 P c6075cc497bd4b37b5c9c4d03e6e6424: Deleting consensus metadata
I20260928 12:31:40.439218 18459 catalog_manager.cc:4841] TS c6075cc497bd4b37b5c9c4d03e6e6424 (127.8.13.2:34923): tablet 4cb87837cb2f49309d1323200b9ca4b3 (table test-table [id=d665a796e77f4bd1a0a8e8776941e87f]) successfully deleted
I20260928 12:31:40.439272 18979 ts_tablet_manager.cc:1918] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260928 12:31:40.439340 18979 log.cc:1196] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/4cb87837cb2f49309d1323200b9ca4b3
I20260928 12:31:40.439893 18979 ts_tablet_manager.cc:1939] T 4cb87837cb2f49309d1323200b9ca4b3 P bcf07c9dadf84eb89d02567016d50ae7: Deleting consensus metadata
I20260928 12:31:40.440349 18460 catalog_manager.cc:4841] TS bcf07c9dadf84eb89d02567016d50ae7 (127.8.13.3:42685): tablet 4cb87837cb2f49309d1323200b9ca4b3 (table test-table [id=d665a796e77f4bd1a0a8e8776941e87f]) successfully deleted
W20260928 12:31:40.443111 18462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:39425 (error 108)
I20260928 12:31:40.443378 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 18671
I20260928 12:31:40.443964 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:31:40.443584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39425 (local address 127.0.0.1:43138)
0928 12:31:40.443878 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 12:31:40.443923 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268}
W20260928 12:31:40.444082 18462 catalog_manager.cc:4584] TS ee9238937bba42cebc6087a3e5d5f4eb (127.8.13.1:39425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cb87837cb2f49309d1323200b9ca4b3: Network error: Client connection negotiation failed: client connection to 127.8.13.1:39425: connect: Connection refused (error 111)
I20260928 12:31:40.449200 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 18805
I20260928 12:31:40.453897 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 18443
2026-09-28T12:31:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2170 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:31:40 dist-test-slave-g31s krb5kdc[18984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:31:40 dist-test-slave-g31s krb5kdc[18984](info): set up 2 sockets
Sep 28 12:31:40 dist-test-slave-g31s krb5kdc[18984](info): commencing operation
krb5kdc: starting...
W20260928 12:31:41.430511 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:31:41 dist-test-slave-g31s krb5kdc[18984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:31:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/rangeradmin_127.8.13.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/rangeradmin_127.8.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/rangerlookup_127.8.13.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/rangerlookup_127.8.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/HTTP_127.8.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/HTTP_127.8.13.20@KRBTEST.COM.keytab.
I20260928 12:31:41.612812 8244 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2Ofqh2/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/postgres -l logfile start
2026-09-28 12:31:44.598 UTC [19014] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 12:31:44.598 UTC [19014] LOG: listening on IPv4 address "127.8.13.20", port 45317
2026-09-28 12:31:44.603 UTC [19014] LOG: listening on Unix socket "/tmp/.s.PGSQL.45317"
2026-09-28 12:31:44.625 UTC [19017] LOG: database system was shut down at 2026-09-28 12:31:42 UTC
2026-09-28 12:31:44.629 UTC [19014] LOG: database system is ready to accept connections
I20260928 12:31:45.514267 8244 mini_postgres.cc:96] Postgres bound to 45317
2026-09-28 12:31:45.520 UTC [19025] FATAL: database "slave" does not exist
127.8.13.20:45317 - accepting connections
I20260928 12:31:45.521215 8244 mini_ranger.cc:161] Starting Ranger...
I20260928 12:31:45.536077 8244 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 12:31:46.118592 8244 mini_ranger.cc:87] Created ranger Postgres database
I20260928 12:31:46.118662 8244 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-admin
2026-09-28 12:31:46,410 [I] DB FLAVOR :POSTGRES
2026-09-28 12:31:46,411 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:31:46,411 [I] Checking connection..
2026-09-28 12:31:46,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 12:31:46,695 [I] Checking connection passed.
2026-09-28 12:31:46,695 [I] --------- Verifying version history table ---------
2026-09-28 12:31:46,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:31:46,952 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:31:46,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:31:47,218 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:31:47,218 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 12:31:47,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 12:31:47.473 UTC [19089] WARNING: there is no transaction in progress
2026-09-28 12:31:47,480 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 12:31:47,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:31:47,759 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 12:31:47,759 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:31:47,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 12:31:48,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 12:31:48,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g31s.c.gcp-upstream.internal','N') ;"
2026-09-28 12:31:48,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:31:48,901 [I] Table x_portal_user does not exist in database ranger
2026-09-28 12:31:48,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:31:49,158 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 12:31:49,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:31:49,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:31:49,701 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 12:31:49,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 12:31:50.878 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:50.902 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:50.924 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:50.946 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:50.971 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:51.003 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:51.094 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:51.107 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:51.128 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:51.143 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:51.159 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:51.180 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:51.194 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:51.205 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:51.214 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:51.223 UTC [19197] WARNING: there is no transaction in progress
2026-09-28 12:31:52,162 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 12:31:52,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 12:31:52,463 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 12:31:52,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:31:52,738 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 12:31:52,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:31:53,003 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 12:31:53,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:31:53,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:31:53,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g31s.c.gcp-upstream.internal';"
2026-09-28 12:31:53,812 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 12:31:53,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 12:31:54,075 [I] DB_PATCHES have already been applied
I20260928 12:31:54.081071 8244 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-admin:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2Ofqh2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 12:31:54.081151 8244 mini_ranger.cc:193] Using host: 127.8.13.20
I20260928 12:31:54.084611 8244 mini_ranger.cc:227] Ranger admin URL: http://127.8.13.20:56509
Sep 28, 2026 12:31:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:31:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:31:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 12:31:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 12:31:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 12:31:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 12:31:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 12:31:56.149399 8244 mini_ranger.cc:160] Time spent starting Ranger: real 10.628s user 0.000s sys 0.012s
Sep 28, 2026 12:32:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 12:32:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 12:32:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 12:32:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 12:32:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 12:32:20.657470 8244 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/rangerkms_127.8.13.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/rangerkms_127.8.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.13.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.13.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.13.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/HTTP_127.8.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.13.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/HTTP_127.8.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 12:32:20 dist-test-slave-g31s krb5kdc[18984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598740, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 12:32:20.752599 8244 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 12:32:20.763556 8244 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 12:32:21.613582 8244 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 12:32:21.613656 8244 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-kms
I20260928 12:32:21.613673 8244 mini_ranger_kms.cc:227] Using postgres at 127.8.13.20:45317
2026-09-28 12:32:22,109 [I] DB FLAVOR :POSTGRES
2026-09-28 12:32:22,110 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:32:22,110 [I] Checking connection
2026-09-28 12:32:22,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 12:32:22,408 [I] connection success
2026-09-28 12:32:22,408 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 12:32:22,408 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 12:32:22,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 12:32:22,679 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 12:32:22,679 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:32:22,679 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 12:32:22,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:45317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 12:32:22,938 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 12:32:23.437839 8244 mini_ranger_kms.cc:310] Created kms service
I20260928 12:32:23.659360 8244 mini_ranger_kms.cc:326] Created kudu user
I20260928 12:32:23.704155 8244 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 12:32:23 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598740, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:32:24.120903 8244 mini_ranger_kms.cc:384] Added ranger policy
I20260928 12:32:24.121027 8244 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-kms:/tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 12:32:24.121047 8244 mini_ranger_kms.cc:240] Using host: 127.8.13.20
I20260928 12:32:24.123901 8244 mini_ranger_kms.cc:276] Ranger KMS PID: 19379
I20260928 12:32:24.124013 8244 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.13.20:44923
Sep 28, 2026 12:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 12:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 12:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 12:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 12:32:26.157697 8244 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.405s user 0.002s sys 0.475s
I20260928 12:32:26.157855 8244 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 12:32:26.157945 8244 mini_ranger_kms.cc:401] 127.8.13.20:44923/kms/v1/keys
Sep 28 12:32:26 dist-test-slave-g31s krb5kdc[18984](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598746, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 12:32:27 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598746, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.20@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
Sep 28 12:32:27 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598746, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.20@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
Sep 28 12:32:29 dist-test-slave-g31s krb5kdc[18984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:32:29.857427 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:43073
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:43073
--ranger_config_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:44923/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:32:29.973436 19429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:32:29.973670 19429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:32:29.973704 19429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:32:29.975186 19429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:32:29.975257 19429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:32:29.975279 19429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:32:29.975294 19429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:32:29.975309 19429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:32:29.977149 19429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.13.20:44923/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:43073
--ranger_config_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:43073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:32:29.977414 19429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:32:29.977626 19429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:32:29.983520 19429 server_base.cc:1034] running on GCE node
W20260928 12:32:29.983659 19434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:32:29.983732 19435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:32:29.983613 19437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:32:29.984043 19429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:32:29.984345 19429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:32:29.985493 19429 hybrid_clock.cc:648] HybridClock initialized: now 1790598749985474 us; error 34 us; skew 500 ppm
Sep 28 12:32:29 dist-test-slave-g31s krb5kdc[18984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598749, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:32:29.993111 19429 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:32:29.993650 19429 webserver.cc:460] Webserver started at http://127.8.13.62:34807/ using document root <none> and password file <none>
I20260928 12:32:29.993920 19429 fs_manager.cc:359] Metadata directory not provided
I20260928 12:32:29.993973 19429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:32:29.994107 19429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:32:29 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598749, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:32:30.041839 19429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "b0f263f5906d4dcca8fb973c806d228e"
format_stamp: "Formatted at 2026-09-28 12:32:30 on dist-test-slave-g31s"
server_key: "02f12dd4083fa0e7ace5e721118f28f4"
server_key_iv: "d835ab93fbbe303519ab0767c79c1627"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:30.042236 19429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "b0f263f5906d4dcca8fb973c806d228e"
format_stamp: "Formatted at 2026-09-28 12:32:30 on dist-test-slave-g31s"
server_key: "02f12dd4083fa0e7ace5e721118f28f4"
server_key_iv: "d835ab93fbbe303519ab0767c79c1627"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:30.043642 19429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:32:30.055451 19444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:32:30.055670 19429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:32:30.055784 19429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "b0f263f5906d4dcca8fb973c806d228e"
format_stamp: "Formatted at 2026-09-28 12:32:30 on dist-test-slave-g31s"
server_key: "02f12dd4083fa0e7ace5e721118f28f4"
server_key_iv: "d835ab93fbbe303519ab0767c79c1627"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:30.055882 19429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:32:30.086710 19429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:32:30.088621 19429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:32:30.088766 19429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:32:30.093314 19429 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:43073
I20260928 12:32:30.093353 19496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:43073 every 8 connection(s)
I20260928 12:32:30.093729 19429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:32:30.094070 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 19429
I20260928 12:32:30.094173 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:32:30.094367 8244 external_mini_cluster.cc:1428] Setting key 28db07fe22158acd86cfcd0b3ba502de
I20260928 12:32:30.094548 19497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:32:30.099156 19497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e: Bootstrap starting.
Sep 28 12:32:30 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:32:30.101367 19497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e: Neither blocks nor log segments found. Creating new log.
I20260928 12:32:30.102888 19497 log.cc:824] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:32:30.104064 19497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e: No bootstrap required, opened a new log
I20260928 12:32:30.104753 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:30.094869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.8.13.62:43073)
0928 12:32:30.095106 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 12:32:30.095114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:32:30.095137 (+ 23us) server_negotiation.cc:407] Connection header received
0928 12:32:30.096162 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:32:30.096170 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:32:30.096223 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:32:30.096359 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:32:30.096968 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:30.097935 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:30.098674 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:30.099105 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:30.099210 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:30.101549 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:32:30.101554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:32:30.101555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:32:30.101569 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:32:30.103095 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:32:30.103496 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:32:30.103500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:32:30.103500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:32:30.103620 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:32:30.103844 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:32:30.103845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:32:30.103846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:32:30.104090 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:32:30.104220 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 12:32:30.104431 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 12:32:30.104545 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260928 12:32:30.107801 19497 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f263f5906d4dcca8fb973c806d228e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43073 } }
I20260928 12:32:30.107998 19497 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:32:30.108031 19497 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f263f5906d4dcca8fb973c806d228e, State: Initialized, Role: FOLLOWER
I20260928 12:32:30.108178 19497 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f263f5906d4dcca8fb973c806d228e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43073 } }
I20260928 12:32:30.108251 19497 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:32:30.108278 19497 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:32:30.108322 19497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:32:30.109467 19497 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f263f5906d4dcca8fb973c806d228e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43073 } }
I20260928 12:32:30.109586 19497 leader_election.cc:302] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0f263f5906d4dcca8fb973c806d228e; no voters:
I20260928 12:32:30.109784 19497 leader_election.cc:288] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:32:30.109872 19502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:32:30.110097 19497 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:32:30.110229 19502 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [term 1 LEADER]: Becoming Leader. State: Replica: b0f263f5906d4dcca8fb973c806d228e, State: Running, Role: LEADER
I20260928 12:32:30.110384 19502 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f263f5906d4dcca8fb973c806d228e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43073 } }
I20260928 12:32:30.110939 19503 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0f263f5906d4dcca8fb973c806d228e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f263f5906d4dcca8fb973c806d228e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43073 } } }
I20260928 12:32:30.111021 19503 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [sys.catalog]: This master's current role is: LEADER
I20260928 12:32:30.111547 19504 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0f263f5906d4dcca8fb973c806d228e. Latest consensus state: current_term: 1 leader_uuid: "b0f263f5906d4dcca8fb973c806d228e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f263f5906d4dcca8fb973c806d228e" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 43073 } } }
I20260928 12:32:30.111621 19504 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e [sys.catalog]: This master's current role is: LEADER
I20260928 12:32:30.112709 19508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:32:30.113346 19508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:32:30.113725 19497 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 12:32:30.115739 19508 catalog_manager.cc:1347] Generated new cluster ID: 7317a94ccc5a49e4aa334ac5198477bd
I20260928 12:32:30.115798 19508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:32:30.132769 19508 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:32:30.133627 19508 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:32:30.143718 19508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e: Generated new TSK 0
I20260928 12:32:30.144025 19508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:32:31.118561 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:44923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43073
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:32:31.266829 19542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:32:31.267151 19542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:32:31.267194 19542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:32:31.269423 19542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:32:31.269501 19542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:32:31.269546 19542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:32:31.271514 19542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.13.20:44923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:32:31.271763 19542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:32:31.272002 19542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:32:31.279556 19548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:32:31.279534 19547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:32:31.279706 19550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:32:31.279959 19542 server_base.cc:1034] running on GCE node
I20260928 12:32:31.280187 19542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:32:31.280478 19542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:32:31.281752 19542 hybrid_clock.cc:648] HybridClock initialized: now 1790598751281690 us; error 34 us; skew 500 ppm
Sep 28 12:32:31 dist-test-slave-g31s krb5kdc[18984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598751, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:32:31.286047 19542 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:32:31.286569 19542 webserver.cc:460] Webserver started at http://127.8.13.1:35453/ using document root <none> and password file <none>
I20260928 12:32:31.286779 19542 fs_manager.cc:359] Metadata directory not provided
I20260928 12:32:31.286828 19542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:32:31.286950 19542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:32:31 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598751, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:32:31.307343 19542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a"
format_stamp: "Formatted at 2026-09-28 12:32:31 on dist-test-slave-g31s"
server_key: "66cf353e62214e819dfc853fbc45bb6b"
server_key_iv: "9126f5582cb12d4a9389ea48e55225b3"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:31.307991 19542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a"
format_stamp: "Formatted at 2026-09-28 12:32:31 on dist-test-slave-g31s"
server_key: "66cf353e62214e819dfc853fbc45bb6b"
server_key_iv: "9126f5582cb12d4a9389ea48e55225b3"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:31.311213 19542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260928 12:32:31.330670 19559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:32:31.330919 19542 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 12:32:31.331008 19542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a"
format_stamp: "Formatted at 2026-09-28 12:32:31 on dist-test-slave-g31s"
server_key: "66cf353e62214e819dfc853fbc45bb6b"
server_key_iv: "9126f5582cb12d4a9389ea48e55225b3"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:31.331101 19542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:32:31.356050 19542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:32:31.358696 19542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:32:31.358865 19542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:32:31.359205 19542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:32:31.359655 19542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:32:31.359702 19542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:32:31.359733 19542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:32:31.359767 19542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:32:31.374881 19542 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:43569
I20260928 12:32:31.375005 19672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:43569 every 8 connection(s)
I20260928 12:32:31.375319 19542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:32:31.376869 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 19542
I20260928 12:32:31.376989 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:32:31.377162 8244 external_mini_cluster.cc:1428] Setting key 4ce51f14480b64abb7d6af15966f9141
Sep 28 12:32:31 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598751, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
I20260928 12:32:31.391105 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.376340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:55547 (local address 127.8.13.62:43073)
0928 12:32:31.377487 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0928 12:32:31.377489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:32:31.377497 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:32:31.377522 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:32:31.377523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:32:31.377535 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:32:31.377600 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:32:31.378283 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.382372 (+ 4089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.384132 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.384455 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.384521 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:31.387346 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:32:31.387348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:32:31.387349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:32:31.387357 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:32:31.389178 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:32:31.389775 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:32:31.389777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:32:31.389777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:32:31.389887 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:32:31.390207 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:32:31.390208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:32:31.390209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:32:31.390382 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:32:31.390454 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:32:31.390878 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 12:32:31.390988 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":64,"threads_started":1}
I20260928 12:32:31.391530 19675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.376252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:43073 (local address 127.8.13.1:55547)
0928 12:32:31.376454 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 12:32:31.376505 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 12:32:31.377183 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:32:31.377702 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:32:31.377706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:32:31.377976 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:32:31.378183 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:31.378188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.382519 (+ 4331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.382521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:31.383982 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:31.383988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.384606 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.384607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:31.384782 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:31.385440 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:32:31.385452 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:32:31.387204 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:32:31.389291 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:32:31.389303 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:32:31.389307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:32:31.389652 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:32:31.389961 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:32:31.389962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:32:31.389962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:32:31.390135 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:32:31.390475 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:32:31.390484 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:32:31.390699 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 12:32:31.390840 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 12:32:31.391094 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260928 12:32:31.392103 19673 heartbeater.cc:347] Connected to a master server at 127.8.13.62:43073
I20260928 12:32:31.392210 19673 heartbeater.cc:464] Registering TS with master...
I20260928 12:32:31.392395 19673 heartbeater.cc:511] Master 127.8.13.62:43073 requested a full tablet report, sending...
I20260928 12:32:31.393137 19461 ts_manager.cc:194] Registered new tserver with Master: 5e6cbdcb80824c8eb6946ce9e1e1e63a (127.8.13.1:43569)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
I20260928 12:32:31.394104 19461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:55547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:32:31.413233 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:44923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43073
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:32:31 dist-test-slave-g31s krb5kdc[18984](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598751, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 12:32:31.575433 19679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:32:31.575873 19679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:32:31.575965 19679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:32:31.578225 19679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:32:31.578382 19679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:32:31.578481 19679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:32:31.581343 19679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.13.20:44923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:32:31.581650 19679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:32:31.582049 19679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:32:31.591310 19686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:32:31.591322 19687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:32:31.592129 19689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:32:31.592687 19679 server_base.cc:1034] running on GCE node
I20260928 12:32:31.592928 19679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:32:31.593228 19679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:32:31.594394 19679 hybrid_clock.cc:648] HybridClock initialized: now 1790598751594380 us; error 29 us; skew 500 ppm
Sep 28 12:32:31 dist-test-slave-g31s krb5kdc[18984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598751, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:32:31.597990 19679 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:32:31.598474 19679 webserver.cc:460] Webserver started at http://127.8.13.2:40489/ using document root <none> and password file <none>
I20260928 12:32:31.598688 19679 fs_manager.cc:359] Metadata directory not provided
I20260928 12:32:31.598737 19679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:32:31.598869 19679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:32:31 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598751, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:32:31.620898 19679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "f29ef556c02048a3a264a35c17065a8b"
format_stamp: "Formatted at 2026-09-28 12:32:31 on dist-test-slave-g31s"
server_key: "828280005569d6bd1ca8cfffd754c2e4"
server_key_iv: "5a31eaa0891f1736985f69d9584e6061"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:31.621363 19679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "f29ef556c02048a3a264a35c17065a8b"
format_stamp: "Formatted at 2026-09-28 12:32:31 on dist-test-slave-g31s"
server_key: "828280005569d6bd1ca8cfffd754c2e4"
server_key_iv: "5a31eaa0891f1736985f69d9584e6061"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:31.623791 19679 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 12:32:31.638540 19697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:32:31.639698 19679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 12:32:31.639811 19679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "f29ef556c02048a3a264a35c17065a8b"
format_stamp: "Formatted at 2026-09-28 12:32:31 on dist-test-slave-g31s"
server_key: "828280005569d6bd1ca8cfffd754c2e4"
server_key_iv: "5a31eaa0891f1736985f69d9584e6061"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:31.639917 19679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:32:31.664060 19679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:32:31.666242 19679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:32:31.666358 19679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:32:31.666602 19679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:32:31.666996 19679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:32:31.667044 19679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:32:31.667068 19679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:32:31.667081 19679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:32:31.675144 19679 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:35179
I20260928 12:32:31.675330 19810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:35179 every 8 connection(s)
I20260928 12:32:31.675650 19679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:32:31.681339 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 19679
I20260928 12:32:31.681450 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:32:31.681639 8244 external_mini_cluster.cc:1428] Setting key a8a8aa2a7f43fc973682e5d5fd7ee8ce
Sep 28 12:32:31 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598751, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:32:31.688364 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.676399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:45283 (local address 127.8.13.62:43073)
0928 12:32:31.676471 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:32:31.676473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:32:31.676661 (+ 188us) server_negotiation.cc:407] Connection header received
0928 12:32:31.677174 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:32:31.677175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:32:31.677189 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:32:31.677280 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:32:31.677854 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.678657 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.679772 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.680091 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.680178 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:31.682950 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:32:31.682952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:32:31.682952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:32:31.682961 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:32:31.684496 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:32:31.685018 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:32:31.685020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:32:31.685020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:32:31.685095 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:32:31.685888 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:32:31.685889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:32:31.685890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:32:31.686059 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:32:31.686107 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:32:31.688120 (+ 2013us) server_negotiation.cc:299] Negotiation successful
0928 12:32:31.688258 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:32:31.689011 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.676362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:43073 (local address 127.8.13.2:45283)
0928 12:32:31.676527 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:32:31.676571 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:32:31.677070 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:32:31.677331 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:32:31.677334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:32:31.677527 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:32:31.677723 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:31.677727 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.678768 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.678769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:31.679681 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:31.679684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.680177 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.680178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:31.680343 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:31.680980 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:32:31.680988 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:32:31.682860 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:32:31.684600 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:32:31.684612 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:32:31.684615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:32:31.684935 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:32:31.685186 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:32:31.685188 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:32:31.685188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:32:31.685375 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:32:31.686157 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:32:31.686165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:32:31.686459 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 12:32:31.688530 (+ 2071us) client_negotiation.cc:240] Negotiation successful
0928 12:32:31.688735 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260928 12:32:31.689656 19811 heartbeater.cc:347] Connected to a master server at 127.8.13.62:43073
I20260928 12:32:31.689781 19811 heartbeater.cc:464] Registering TS with master...
I20260928 12:32:31.690024 19811 heartbeater.cc:511] Master 127.8.13.62:43073 requested a full tablet report, sending...
I20260928 12:32:31.690456 19461 ts_manager.cc:194] Registered new tserver with Master: f29ef556c02048a3a264a35c17065a8b (127.8.13.2:35179)
I20260928 12:32:31.690824 19461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:45283
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:32:31.716755 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:44923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43073
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:32:31.855580 19816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:32:31.855825 19816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:32:31.855859 19816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:32:31.857164 19816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:32:31.857216 19816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:32:31.857255 19816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:32:31.858989 19816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.13.20:44923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:43073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:32:31.859256 19816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:32:31.859539 19816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:32:31.866185 19824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:32:31.866209 19822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:32:31.866400 19816 server_base.cc:1034] running on GCE node
W20260928 12:32:31.866230 19821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:32:31.866626 19816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:32:31.866931 19816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:32:31.868103 19816 hybrid_clock.cc:648] HybridClock initialized: now 1790598751868089 us; error 27 us; skew 500 ppm
Sep 28 12:32:31 dist-test-slave-g31s krb5kdc[18984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598751, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:32:31.871863 19816 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:32:31.872340 19816 webserver.cc:460] Webserver started at http://127.8.13.3:34049/ using document root <none> and password file <none>
I20260928 12:32:31.872574 19816 fs_manager.cc:359] Metadata directory not provided
I20260928 12:32:31.872627 19816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:32:31.872766 19816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:32:31 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598751, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:32:31.887727 19816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "b072876e83fc4779bf1a5474efebc6a9"
format_stamp: "Formatted at 2026-09-28 12:32:31 on dist-test-slave-g31s"
server_key: "9b67c97f5e2a5e9fe323ba2e19ce1578"
server_key_iv: "47de784f737a1d02c6d44eb1a0c48bb7"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:31.888320 19816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "b072876e83fc4779bf1a5474efebc6a9"
format_stamp: "Formatted at 2026-09-28 12:32:31 on dist-test-slave-g31s"
server_key: "9b67c97f5e2a5e9fe323ba2e19ce1578"
server_key_iv: "47de784f737a1d02c6d44eb1a0c48bb7"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:31.891083 19816 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 12:32:31.916577 19831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:32:31.917164 19816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 12:32:31.917272 19816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "b072876e83fc4779bf1a5474efebc6a9"
format_stamp: "Formatted at 2026-09-28 12:32:31 on dist-test-slave-g31s"
server_key: "9b67c97f5e2a5e9fe323ba2e19ce1578"
server_key_iv: "47de784f737a1d02c6d44eb1a0c48bb7"
server_key_version: "kuduclusterkey@0"
I20260928 12:32:31.917385 19816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:32:31.930419 19816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:32:31.932459 19816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:32:31.932591 19816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:32:31.932809 19816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:32:31.933152 19816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:32:31.933197 19816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:32:31.933229 19816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:32:31.933250 19816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:32:31.946557 19816 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:46015
I20260928 12:32:31.946777 19944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:46015 every 8 connection(s)
I20260928 12:32:31.947232 19816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:32:31.951439 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 19816
I20260928 12:32:31.951551 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:32:31.951725 8244 external_mini_cluster.cc:1428] Setting key b14de355740074b5c909900433e43f52
Sep 28 12:32:31 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598751, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:32:31.963889 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.948325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:58761 (local address 127.8.13.62:43073)
0928 12:32:31.948381 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:32:31.948383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:32:31.948903 (+ 520us) server_negotiation.cc:407] Connection header received
0928 12:32:31.950094 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:32:31.950095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:32:31.950109 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:32:31.950141 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:32:31.951457 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.952266 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.954052 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.954354 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.954402 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:31.957873 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:32:31.957875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:32:31.957876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:32:31.957885 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:32:31.959603 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:32:31.962531 (+ 2928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:32:31.962534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:32:31.962534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:32:31.962616 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:32:31.963002 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:32:31.963003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:32:31.963004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:32:31.963160 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:32:31.963215 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:32:31.963685 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 12:32:31.963780 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:32:31.964497 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.948331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:43073 (local address 127.8.13.3:58761)
0928 12:32:31.948701 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 12:32:31.948752 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 12:32:31.949949 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:32:31.950339 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:32:31.950344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:32:31.950850 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:32:31.951343 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:31.951361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.952391 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.952392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:31.953931 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:31.953935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.954505 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.954507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:31.954750 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:31.955577 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:32:31.955588 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:32:31.957711 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:32:31.961942 (+ 4231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:32:31.961956 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:32:31.961960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:32:31.962357 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:32:31.962710 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:32:31.962711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:32:31.962711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:32:31.962904 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:32:31.963251 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:32:31.963260 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:32:31.963523 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 12:32:31.964001 (+ 478us) client_negotiation.cc:240] Negotiation successful
0928 12:32:31.964193 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":206,"threads_started":1}
I20260928 12:32:31.965174 19945 heartbeater.cc:347] Connected to a master server at 127.8.13.62:43073
I20260928 12:32:31.965287 19945 heartbeater.cc:464] Registering TS with master...
I20260928 12:32:31.965507 19945 heartbeater.cc:511] Master 127.8.13.62:43073 requested a full tablet report, sending...
I20260928 12:32:31.965920 19461 ts_manager.cc:194] Registered new tserver with Master: b072876e83fc4779bf1a5474efebc6a9 (127.8.13.3:46015)
I20260928 12:32:31.966404 19461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:58761
I20260928 12:32:31.974225 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:32:31.985749 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.976092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.8.13.62:43073)
0928 12:32:31.976136 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:32:31.976138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:32:31.977553 (+ 1415us) server_negotiation.cc:407] Connection header received
0928 12:32:31.977641 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:32:31.977642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:32:31.977658 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:32:31.977728 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:32:31.978570 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.979363 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.980539 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.980869 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.980950 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:31.982560 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:32:31.982562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:32:31.982562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:32:31.982571 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:32:31.984306 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:32:31.984760 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:32:31.984762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:32:31.984763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:32:31.984831 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:32:31.985051 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:32:31.985053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:32:31.985053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:32:31.985221 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:32:31.985335 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 12:32:31.985550 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 12:32:31.985647 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:32:31.988561 19461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:32:31.991307 19461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:32:32.000288 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.994841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:35179 (local address 127.0.0.1:48410)
0928 12:32:31.995614 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 12:32:31.995622 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:32:31.995662 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:32:31.995988 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:32:31.995991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:32:31.995994 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:32:31.996108 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:31.996110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.997418 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.997419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:31.998801 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:31.998805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.999944 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.999946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:32.000034 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:32.000048 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:32:32.000117 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 12:32:32.000178 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":67,"threads_started":1}
I20260928 12:32:32.000875 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.995043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.8.13.2:35179)
0928 12:32:31.995216 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 12:32:31.995222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:32:31.995721 (+ 499us) server_negotiation.cc:407] Connection header received
0928 12:32:31.995783 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:32:31.995785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:32:31.995804 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:32:31.995879 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:32:31.996193 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.997308 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.998929 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:31.999751 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:31.999827 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:31.999907 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:32:32.000746 (+ 839us) server_negotiation.cc:299] Negotiation successful
0928 12:32:32.000794 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260928 12:32:32.001916 19745 tablet_service.cc:1467] Processing CreateTablet for tablet 3728c2388fd248ce8bab60e2e6009930 (DEFAULT_TABLE table=test-table [id=2c8c93e17ac140da84d6f1bc0df21c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:32:32.002318 19745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3728c2388fd248ce8bab60e2e6009930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:32:32.006192 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.994553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:46015 (local address 127.0.0.1:51882)
0928 12:32:31.995500 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0928 12:32:31.995509 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:32:31.995545 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:32:32.001663 (+ 6118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:32:32.001665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:32:32.001667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:32:32.001717 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:32.001718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.003435 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.003436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:32.004573 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:32.004575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.005856 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.005858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:32.006022 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:32.006028 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:32:32.006057 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 12:32:32.006105 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":112,"threads_started":1}
I20260928 12:32:32.006599 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.995053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.8.13.3:46015)
0928 12:32:32.001481 (+ 6428us) server_negotiation.cc:206] Beginning negotiation
0928 12:32:32.001487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:32:32.001498 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:32:32.001537 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:32:32.001539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:32:32.001556 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:32:32.001780 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:32:32.001882 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.003341 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.004689 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.005723 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.006372 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:32.006459 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:32:32.006484 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 12:32:32.006525 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6384,"thread_start_us":76,"threads_started":1}
I20260928 12:32:32.007198 19860 tablet_service.cc:1467] Processing CreateTablet for tablet 3728c2388fd248ce8bab60e2e6009930 (DEFAULT_TABLE table=test-table [id=2c8c93e17ac140da84d6f1bc0df21c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:32:32.007532 19860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3728c2388fd248ce8bab60e2e6009930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:32:32.014056 19963 tablet_bootstrap.cc:492] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b: Bootstrap starting.
I20260928 12:32:32.015013 19963 tablet_bootstrap.cc:654] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b: Neither blocks nor log segments found. Creating new log.
I20260928 12:32:32.015277 19963 log.cc:824] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:32:32.015378 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.994552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:43569 (local address 127.0.0.1:44124)
0928 12:32:31.995276 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 12:32:31.995324 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:32:31.995395 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:32:32.011390 (+ 15995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:32:32.011391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:32:32.011394 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:32:32.011461 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:32.011463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.012782 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.012783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:32.013953 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:32.013956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.015128 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.015129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:32.015207 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:32.015213 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:32:32.015252 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 12:32:32.015297 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":106,"threads_started":1}
I20260928 12:32:32.015753 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:31.994841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.8.13.1:43569)
0928 12:32:32.011155 (+ 16314us) server_negotiation.cc:206] Beginning negotiation
0928 12:32:32.011160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:32:32.011172 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:32:32.011208 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:32:32.011210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:32:32.011228 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:32:32.011532 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:32:32.011598 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.012686 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.014166 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.015033 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.015543 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:32.015616 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:32:32.015642 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 12:32:32.015684 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16273,"thread_start_us":53,"threads_started":1}
I20260928 12:32:32.016115 19964 tablet_bootstrap.cc:492] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9: Bootstrap starting.
I20260928 12:32:32.016435 19607 tablet_service.cc:1467] Processing CreateTablet for tablet 3728c2388fd248ce8bab60e2e6009930 (DEFAULT_TABLE table=test-table [id=2c8c93e17ac140da84d6f1bc0df21c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:32:32.016768 19607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3728c2388fd248ce8bab60e2e6009930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:32:32.016991 19964 tablet_bootstrap.cc:654] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9: Neither blocks nor log segments found. Creating new log.
I20260928 12:32:32.017282 19964 log.cc:824] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:32:32.017987 19964 tablet_bootstrap.cc:492] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9: No bootstrap required, opened a new log
I20260928 12:32:32.018050 19964 ts_tablet_manager.cc:1397] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:32:32.020732 19964 raft_consensus.cc:348] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b072876e83fc4779bf1a5474efebc6a9" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46015 } } peers { permanent_uuid: "f29ef556c02048a3a264a35c17065a8b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35179 } } peers { permanent_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43569 } }
I20260928 12:32:32.020906 19964 raft_consensus.cc:374] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:32:32.020944 19964 raft_consensus.cc:729] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b072876e83fc4779bf1a5474efebc6a9, State: Initialized, Role: FOLLOWER
I20260928 12:32:32.021435 19964 consensus_queue.cc:260] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b072876e83fc4779bf1a5474efebc6a9" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46015 } } peers { permanent_uuid: "f29ef556c02048a3a264a35c17065a8b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35179 } } peers { permanent_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43569 } }
I20260928 12:32:32.021831 19964 ts_tablet_manager.cc:1428] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 12:32:32.022047 19945 heartbeater.cc:503] Master 127.8.13.62:43073 was elected leader, sending a full tablet report...
I20260928 12:32:32.022436 19967 tablet_bootstrap.cc:492] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a: Bootstrap starting.
I20260928 12:32:32.024631 19967 tablet_bootstrap.cc:654] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a: Neither blocks nor log segments found. Creating new log.
I20260928 12:32:32.025559 19967 log.cc:824] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:32:32.025678 19963 tablet_bootstrap.cc:492] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b: No bootstrap required, opened a new log
I20260928 12:32:32.025751 19963 ts_tablet_manager.cc:1397] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 12:32:32.026510 19967 tablet_bootstrap.cc:492] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a: No bootstrap required, opened a new log
I20260928 12:32:32.026566 19967 ts_tablet_manager.cc:1397] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 12:32:32.030220 19963 raft_consensus.cc:348] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b072876e83fc4779bf1a5474efebc6a9" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46015 } } peers { permanent_uuid: "f29ef556c02048a3a264a35c17065a8b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35179 } } peers { permanent_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43569 } }
I20260928 12:32:32.030438 19963 raft_consensus.cc:374] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:32:32.030485 19963 raft_consensus.cc:729] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29ef556c02048a3a264a35c17065a8b, State: Initialized, Role: FOLLOWER
I20260928 12:32:32.030604 19963 consensus_queue.cc:260] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b072876e83fc4779bf1a5474efebc6a9" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46015 } } peers { permanent_uuid: "f29ef556c02048a3a264a35c17065a8b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35179 } } peers { permanent_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43569 } }
I20260928 12:32:32.030946 19963 ts_tablet_manager.cc:1428] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260928 12:32:32.031085 19811 heartbeater.cc:503] Master 127.8.13.62:43073 was elected leader, sending a full tablet report...
I20260928 12:32:32.031484 19967 raft_consensus.cc:348] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b072876e83fc4779bf1a5474efebc6a9" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46015 } } peers { permanent_uuid: "f29ef556c02048a3a264a35c17065a8b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35179 } } peers { permanent_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43569 } }
I20260928 12:32:32.031638 19967 raft_consensus.cc:374] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:32:32.031665 19967 raft_consensus.cc:729] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6cbdcb80824c8eb6946ce9e1e1e63a, State: Initialized, Role: FOLLOWER
I20260928 12:32:32.031788 19967 consensus_queue.cc:260] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b072876e83fc4779bf1a5474efebc6a9" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46015 } } peers { permanent_uuid: "f29ef556c02048a3a264a35c17065a8b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35179 } } peers { permanent_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43569 } }
I20260928 12:32:32.032095 19967 ts_tablet_manager.cc:1428] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260928 12:32:32.032202 19673 heartbeater.cc:503] Master 127.8.13.62:43073 was elected leader, sending a full tablet report...
Sep 28 12:32:32 dist-test-slave-g31s krb5kdc[18984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598751, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:32:32.065891 19970 raft_consensus.cc:482] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:32:32.066041 19970 raft_consensus.cc:504] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b072876e83fc4779bf1a5474efebc6a9" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46015 } } peers { permanent_uuid: "f29ef556c02048a3a264a35c17065a8b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35179 } } peers { permanent_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43569 } }
I20260928 12:32:32.066533 19970 leader_election.cc:288] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b072876e83fc4779bf1a5474efebc6a9 (127.8.13.3:46015), 5e6cbdcb80824c8eb6946ce9e1e1e63a (127.8.13.1:43569)
I20260928 12:32:32.071228 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:32.066719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:46015 (local address 127.8.13.2:59779)
0928 12:32:32.067113 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 12:32:32.067126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:32:32.067182 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:32:32.067385 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:32:32.067386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:32:32.067388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:32:32.067588 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:32.067590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.068723 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.068724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:32.070009 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:32.070013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.070896 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.070897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:32.071043 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:32.071047 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:32:32.071096 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:32:32.071142 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375}
I20260928 12:32:32.071563 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:32.066791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:59779 (local address 127.8.13.3:46015)
0928 12:32:32.066837 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:32:32.066839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:32:32.067180 (+ 341us) server_negotiation.cc:407] Connection header received
0928 12:32:32.067247 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:32:32.067248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:32:32.067260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:32:32.067327 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:32:32.067674 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.068491 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.070111 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.070782 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.070848 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:32.070861 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:32:32.071448 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 12:32:32.071496 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:32:32.072000 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3728c2388fd248ce8bab60e2e6009930" candidate_uuid: "f29ef556c02048a3a264a35c17065a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b072876e83fc4779bf1a5474efebc6a9" is_pre_election: true
I20260928 12:32:32.072183 19899 raft_consensus.cc:2454] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29ef556c02048a3a264a35c17065a8b in term 0.
I20260928 12:32:32.072511 19700 leader_election.cc:302] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b072876e83fc4779bf1a5474efebc6a9, f29ef556c02048a3a264a35c17065a8b; no voters:
I20260928 12:32:32.072690 19970 raft_consensus.cc:2781] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:32:32.072741 19970 raft_consensus.cc:482] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:32:32.072767 19970 raft_consensus.cc:3037] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:32:32.073725 19970 raft_consensus.cc:504] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b072876e83fc4779bf1a5474efebc6a9" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46015 } } peers { permanent_uuid: "f29ef556c02048a3a264a35c17065a8b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35179 } } peers { permanent_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43569 } }
I20260928 12:32:32.074040 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3728c2388fd248ce8bab60e2e6009930" candidate_uuid: "f29ef556c02048a3a264a35c17065a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b072876e83fc4779bf1a5474efebc6a9"
I20260928 12:32:32.074149 19899 raft_consensus.cc:3037] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:32:32.075007 19899 raft_consensus.cc:2454] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29ef556c02048a3a264a35c17065a8b in term 1.
I20260928 12:32:32.075418 19700 leader_election.cc:302] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b072876e83fc4779bf1a5474efebc6a9, f29ef556c02048a3a264a35c17065a8b; no voters:
I20260928 12:32:32.075734 19970 leader_election.cc:288] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [CANDIDATE]: Term 1 election: Requested vote from peers b072876e83fc4779bf1a5474efebc6a9 (127.8.13.3:46015), 5e6cbdcb80824c8eb6946ce9e1e1e63a (127.8.13.1:43569)
I20260928 12:32:32.075909 19970 raft_consensus.cc:2781] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:32:32.075958 19970 raft_consensus.cc:686] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 1 LEADER]: Becoming Leader. State: Replica: f29ef556c02048a3a264a35c17065a8b, State: Running, Role: LEADER
I20260928 12:32:32.075966 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:32.067025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:40327 (local address 127.8.13.1:43569)
0928 12:32:32.067062 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:32:32.067064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:32:32.070957 (+ 3893us) server_negotiation.cc:407] Connection header received
0928 12:32:32.071070 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:32:32.071072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:32:32.071086 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:32:32.071116 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:32:32.071812 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.072462 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.074472 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.075087 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.075137 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:32.075150 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:32:32.075845 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 12:32:32.075895 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:32:32.076043 19970 consensus_queue.cc:237] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b072876e83fc4779bf1a5474efebc6a9" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46015 } } peers { permanent_uuid: "f29ef556c02048a3a264a35c17065a8b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35179 } } peers { permanent_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43569 } }
I20260928 12:32:32.077286 19461 catalog_manager.cc:5495] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b reported cstate change: term changed from 0 to 1, leader changed from <none> to f29ef556c02048a3a264a35c17065a8b (127.8.13.2). New cstate: current_term: 1 leader_uuid: "f29ef556c02048a3a264a35c17065a8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b072876e83fc4779bf1a5474efebc6a9" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29ef556c02048a3a264a35c17065a8b" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43569 } health_report { overall_health: UNKNOWN } } }
I20260928 12:32:32.076465 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:32.066961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:43569 (local address 127.8.13.2:40327)
0928 12:32:32.070871 (+ 3910us) negotiation.cc:107] Waiting for socket to connect
0928 12:32:32.070887 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:32:32.070991 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:32:32.071184 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:32:32.071184 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:32:32.071186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:32:32.071266 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:32.071268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.072821 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.072822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:32.074323 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:32:32.074327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.075462 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.075464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:32:32.075589 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:32.075595 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:32:32.076315 (+ 720us) client_negotiation.cc:240] Negotiation successful
0928 12:32:32.076369 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3881,"thread_start_us":107,"threads_started":1}
I20260928 12:32:32.078029 19627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3728c2388fd248ce8bab60e2e6009930" candidate_uuid: "f29ef556c02048a3a264a35c17065a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" is_pre_election: true
I20260928 12:32:32.078187 19627 raft_consensus.cc:2454] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29ef556c02048a3a264a35c17065a8b in term 0.
I20260928 12:32:32.078609 19627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3728c2388fd248ce8bab60e2e6009930" candidate_uuid: "f29ef556c02048a3a264a35c17065a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a"
I20260928 12:32:32.078693 19627 raft_consensus.cc:3037] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:32:32.079658 19627 raft_consensus.cc:2454] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29ef556c02048a3a264a35c17065a8b in term 1.
I20260928 12:32:32.099929 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:32.096412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.8.13.2:35179)
0928 12:32:32.096480 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:32:32.096482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:32:32.096489 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:32:32.096530 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:32:32.096531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:32:32.096556 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:32:32.096615 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:32:32.096812 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.097574 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.098731 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:32:32.099046 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:32:32.099103 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:32:32.099318 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:32:32.099466 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:32:32.099546 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:32:32.099800 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 12:32:32.099825 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:32:32.102035 19627 raft_consensus.cc:1261] T 3728c2388fd248ce8bab60e2e6009930 P 5e6cbdcb80824c8eb6946ce9e1e1e63a [term 1 FOLLOWER]: Refusing update from remote peer f29ef556c02048a3a264a35c17065a8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:32:32.102519 19970 consensus_queue.cc:1035] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e6cbdcb80824c8eb6946ce9e1e1e63a" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:32:32.103873 19899 raft_consensus.cc:1261] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9 [term 1 FOLLOWER]: Refusing update from remote peer f29ef556c02048a3a264a35c17065a8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:32:32.104367 19973 consensus_queue.cc:1035] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b072876e83fc4779bf1a5474efebc6a9" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:32:32.107288 19976 mvcc.cc:205] Tried to move back new op lower bound from 7334292488606601216 to 7334292488504111104. Current Snapshot: MvccSnapshot[applied={T|T < 7334292488606601216}]
I20260928 12:32:32.108227 19979 mvcc.cc:205] Tried to move back new op lower bound from 7334292488606601216 to 7334292488504111104. Current Snapshot: MvccSnapshot[applied={T|T < 7334292488606601216}]
I20260928 12:32:32.112061 19977 mvcc.cc:205] Tried to move back new op lower bound from 7334292488606601216 to 7334292488504111104. Current Snapshot: MvccSnapshot[applied={T|T < 7334292488606601216}]
I20260928 12:32:32.114099 19460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:32:32.114210 19460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:32:32.114964 19460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0f263f5906d4dcca8fb973c806d228e: Sending DeleteTablet for 3 replicas of tablet 3728c2388fd248ce8bab60e2e6009930
I20260928 12:32:32.115326 19745 tablet_service.cc:1514] Processing DeleteTablet for tablet 3728c2388fd248ce8bab60e2e6009930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:32:32 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:48410
I20260928 12:32:32.115631 19607 tablet_service.cc:1514] Processing DeleteTablet for tablet 3728c2388fd248ce8bab60e2e6009930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:32:32 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:44124
I20260928 12:32:32.116000 19860 tablet_service.cc:1514] Processing DeleteTablet for tablet 3728c2388fd248ce8bab60e2e6009930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:32:32 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:51882
I20260928 12:32:32.116564 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 19542
I20260928 12:32:32.118467 19988 tablet_replica.cc:330] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9: stopping tablet replica
I20260928 12:32:32.118633 19988 raft_consensus.cc:2229] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:32:32.118816 19988 raft_consensus.cc:2258] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:32:32.119354 19988 ts_tablet_manager.cc:1905] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:32:32.119522 19986 tablet_replica.cc:330] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b: stopping tablet replica
I20260928 12:32:32.119688 19986 raft_consensus.cc:2229] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:32:32.119885 19986 raft_consensus.cc:2258] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:32:32.121490 19986 ts_tablet_manager.cc:1905] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:32:32.122512 19988 ts_tablet_manager.cc:1918] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:32:32.122666 19988 log.cc:1196] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/3728c2388fd248ce8bab60e2e6009930
I20260928 12:32:32.123039 19988 ts_tablet_manager.cc:1939] T 3728c2388fd248ce8bab60e2e6009930 P b072876e83fc4779bf1a5474efebc6a9: Deleting consensus metadata
I20260928 12:32:32.123503 19447 catalog_manager.cc:4841] TS b072876e83fc4779bf1a5474efebc6a9 (127.8.13.3:46015): tablet 3728c2388fd248ce8bab60e2e6009930 (table test-table [id=2c8c93e17ac140da84d6f1bc0df21c20]) successfully deleted
I20260928 12:32:32.123531 19986 ts_tablet_manager.cc:1918] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:32:32.123586 19986 log.cc:1196] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/3728c2388fd248ce8bab60e2e6009930
I20260928 12:32:32.123875 19986 ts_tablet_manager.cc:1939] T 3728c2388fd248ce8bab60e2e6009930 P f29ef556c02048a3a264a35c17065a8b: Deleting consensus metadata
I20260928 12:32:32.124221 19446 catalog_manager.cc:4841] TS f29ef556c02048a3a264a35c17065a8b (127.8.13.2:35179): tablet 3728c2388fd248ce8bab60e2e6009930 (table test-table [id=2c8c93e17ac140da84d6f1bc0df21c20]) successfully deleted
W20260928 12:32:32.134128 19445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:43569 (error 108)
I20260928 12:32:32.138023 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 19679
I20260928 12:32:32.141664 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:32:32.141483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:43569 (local address 127.0.0.1:44136)
0928 12:32:32.141549 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 12:32:32.141613 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:32:32.142026 19445 catalog_manager.cc:4584] TS 5e6cbdcb80824c8eb6946ce9e1e1e63a (127.8.13.1:43569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3728c2388fd248ce8bab60e2e6009930: Network error: Client connection negotiation failed: client connection to 127.8.13.1:43569: connect: Connection refused (error 111)
I20260928 12:32:32.146448 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 19816
I20260928 12:32:32.152927 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 19429
I20260928 12:32:32.159200 8244 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 12:32:32.503517 8244 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 12:32:32.503628 8244 mini_ranger.cc:66] Stopping Ranger...
I20260928 12:32:32.838384 8244 mini_ranger.cc:68] Stopped Ranger
2026-09-28 12:32:32.838 UTC [19014] LOG: received smart shutdown request
2026-09-28 12:32:32.842 UTC [19014] LOG: background worker "logical replication launcher" (PID 19023) exited with exit code 1
2026-09-28 12:32:32.842 UTC [19018] LOG: shutting down
2026-09-28 12:32:32.981 UTC [19014] LOG: database system is shut down
2026-09-28T12:32:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52616 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:32:33 dist-test-slave-g31s krb5kdc[19998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:32:33 dist-test-slave-g31s krb5kdc[19998](info): set up 2 sockets
Sep 28 12:32:33 dist-test-slave-g31s krb5kdc[19998](info): commencing operation
krb5kdc: starting...
W20260928 12:32:34.017730 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:32:34 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:32:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/rangeradmin_127.8.13.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/rangeradmin_127.8.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/rangerlookup_127.8.13.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/rangerlookup_127.8.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/HTTP_127.8.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/HTTP_127.8.13.20@KRBTEST.COM.keytab.
I20260928 12:32:34.173269 8244 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2Ofqh2/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/postgres -l logfile start
2026-09-28 12:32:36.525 UTC [20028] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 12:32:36.525 UTC [20028] LOG: listening on IPv4 address "127.8.13.20", port 35991
2026-09-28 12:32:36.528 UTC [20028] LOG: listening on Unix socket "/tmp/.s.PGSQL.35991"
2026-09-28 12:32:36.548 UTC [20031] LOG: database system was shut down at 2026-09-28 12:32:35 UTC
2026-09-28 12:32:36.551 UTC [20028] LOG: database system is ready to accept connections
I20260928 12:32:37.417325 8244 mini_postgres.cc:96] Postgres bound to 35991
2026-09-28 12:32:37.422 UTC [20039] FATAL: database "slave" does not exist
127.8.13.20:35991 - accepting connections
I20260928 12:32:37.423113 8244 mini_ranger.cc:161] Starting Ranger...
I20260928 12:32:37.436816 8244 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 12:32:37.886650 8244 mini_ranger.cc:87] Created ranger Postgres database
I20260928 12:32:37.886721 8244 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-admin
2026-09-28 12:32:38,147 [I] DB FLAVOR :POSTGRES
2026-09-28 12:32:38,147 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:32:38,147 [I] Checking connection..
2026-09-28 12:32:38,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 12:32:38,404 [I] Checking connection passed.
2026-09-28 12:32:38,405 [I] --------- Verifying version history table ---------
2026-09-28 12:32:38,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:32:38,667 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:32:38,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:32:38,897 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:32:38,897 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 12:32:38,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 12:32:39.138 UTC [20103] WARNING: there is no transaction in progress
2026-09-28 12:32:39,143 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 12:32:39,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:32:39,407 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 12:32:39,408 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:32:39,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 12:32:39,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 12:32:39,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g31s.c.gcp-upstream.internal','N') ;"
2026-09-28 12:32:40,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:32:40,397 [I] Table x_portal_user does not exist in database ranger
2026-09-28 12:32:40,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:32:40,641 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 12:32:40,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:32:40,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:32:41,151 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 12:32:41,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 12:32:42.137 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.153 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.169 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.184 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.202 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.223 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.291 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.298 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.312 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.323 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.334 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.349 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.360 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.367 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.375 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:42.382 UTC [20211] WARNING: there is no transaction in progress
2026-09-28 12:32:43,039 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 12:32:43,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 12:32:43,292 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 12:32:43,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:32:43,528 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 12:32:43,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:32:43,793 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 12:32:43,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:32:44,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:32:44,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g31s.c.gcp-upstream.internal';"
2026-09-28 12:32:44,522 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 12:32:44,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 12:32:44,774 [I] DB_PATCHES have already been applied
I20260928 12:32:44.780448 8244 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-admin:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2Ofqh2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 12:32:44.780524 8244 mini_ranger.cc:193] Using host: 127.8.13.20
I20260928 12:32:44.783527 8244 mini_ranger.cc:227] Ranger admin URL: http://127.8.13.20:36393
Sep 28, 2026 12:32:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:32:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:32:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 12:32:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 12:32:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 12:32:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 12:32:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 12:32:45.807554 8244 mini_ranger.cc:160] Time spent starting Ranger: real 8.384s user 0.001s sys 0.007s
Sep 28, 2026 12:33:06 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 12:33:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 12:33:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 12:33:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 12:33:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 12:33:07.818800 8244 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/rangerkms_127.8.13.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/rangerkms_127.8.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.13.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.13.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.13.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/HTTP_127.8.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.13.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/HTTP_127.8.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 12:33:07 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598787, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 12:33:07.916148 8244 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 12:33:07.926044 8244 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 12:33:08.855396 8244 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 12:33:08.855469 8244 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-kms
I20260928 12:33:08.855487 8244 mini_ranger_kms.cc:227] Using postgres at 127.8.13.20:35991
2026-09-28 12:33:09,014 [I] DB FLAVOR :POSTGRES
2026-09-28 12:33:09,014 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:33:09,014 [I] Checking connection
2026-09-28 12:33:09,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 12:33:09,289 [I] connection success
2026-09-28 12:33:09,290 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 12:33:09,290 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 12:33:09,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 12:33:09,605 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 12:33:09,606 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:33:09,606 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 12:33:09,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.20:35991/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 12:33:09,926 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 12:33:10.396283 8244 mini_ranger_kms.cc:310] Created kms service
I20260928 12:33:10.598963 8244 mini_ranger_kms.cc:326] Created kudu user
I20260928 12:33:10.649780 8244 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 12:33:10 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598787, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:33:11.102319 8244 mini_ranger_kms.cc:384] Added ranger policy
I20260928 12:33:11.102591 8244 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-kms:/tmp/dist-test-task2Ofqh2/build/release/bin/postgresql.jar:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2Ofqh2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 12:33:11.102622 8244 mini_ranger_kms.cc:240] Using host: 127.8.13.20
I20260928 12:33:11.106084 8244 mini_ranger_kms.cc:276] Ranger KMS PID: 20391
I20260928 12:33:11.106169 8244 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.13.20:35599
Sep 28, 2026 12:33:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:33:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:33:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 12:33:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 12:33:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 12:33:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 12:33:13.169059 8244 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.253s user 0.006s sys 0.136s
I20260928 12:33:13.169157 8244 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 12:33:13.169204 8244 mini_ranger_kms.cc:401] 127.8.13.20:35599/kms/v1/keys
Sep 28 12:33:13 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598793, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 12:33:13 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598793, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.20@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
Sep 28 12:33:14 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598793, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.20@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
Sep 28 12:33:16 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:33:16.428294 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33861
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:33861
--ranger_config_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:35599/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:16.508213 20442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:16.508435 20442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:16.508464 20442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:16.509639 20442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:33:16.509684 20442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:16.509702 20442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:16.509718 20442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:33:16.509733 20442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:33:16.511344 20442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.13.20:35599/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:33861
--ranger_config_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:16.511586 20442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:16.511790 20442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:16.516453 20448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:16.516450 20447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:16.516553 20450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:16.516726 20442 server_base.cc:1034] running on GCE node
I20260928 12:33:16.517014 20442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:16.517299 20442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:16.518435 20442 hybrid_clock.cc:648] HybridClock initialized: now 1790598796518415 us; error 38 us; skew 500 ppm
Sep 28 12:33:16 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598796, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:16.521005 20442 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:33:16.521473 20442 webserver.cc:460] Webserver started at http://127.8.13.62:34169/ using document root <none> and password file <none>
I20260928 12:33:16.521680 20442 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:16.521719 20442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:16.521870 20442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:33:16 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598796, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:33:16.566817 20442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "afa29a2068c441fcadb29fcfc937a216"
format_stamp: "Formatted at 2026-09-28 12:33:16 on dist-test-slave-g31s"
server_key: "d453e0a639e6546939a4ebe26aa1ee43"
server_key_iv: "d1d36b0b4cd887829174e94a865c9b0e"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:16.567189 20442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "afa29a2068c441fcadb29fcfc937a216"
format_stamp: "Formatted at 2026-09-28 12:33:16 on dist-test-slave-g31s"
server_key: "d453e0a639e6546939a4ebe26aa1ee43"
server_key_iv: "d1d36b0b4cd887829174e94a865c9b0e"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:16.568686 20442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:33:16.583617 20457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:16.583926 20442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:33:16.584033 20442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "afa29a2068c441fcadb29fcfc937a216"
format_stamp: "Formatted at 2026-09-28 12:33:16 on dist-test-slave-g31s"
server_key: "d453e0a639e6546939a4ebe26aa1ee43"
server_key_iv: "d1d36b0b4cd887829174e94a865c9b0e"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:16.584129 20442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:16.605073 20442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:16.607163 20442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:16.607311 20442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:16.612177 20442 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:33861
I20260928 12:33:16.612241 20509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:33861 every 8 connection(s)
I20260928 12:33:16.612618 20442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:33:16.613339 20510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:16.614288 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 20442
I20260928 12:33:16.614391 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:33:16.614569 8244 external_mini_cluster.cc:1428] Setting key fe79ca8c13cc7e43138ec1c8408bc469
I20260928 12:33:16.616374 20510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216: Bootstrap starting.
I20260928 12:33:16.617105 20510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:16.617475 20510 log.cc:824] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:16.618268 20510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216: No bootstrap required, opened a new log
I20260928 12:33:16.620139 20510 raft_consensus.cc:348] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } }
I20260928 12:33:16.620328 20510 raft_consensus.cc:374] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:16.620363 20510 raft_consensus.cc:729] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa29a2068c441fcadb29fcfc937a216, State: Initialized, Role: FOLLOWER
I20260928 12:33:16.620517 20510 consensus_queue.cc:260] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } }
I20260928 12:33:16.620615 20510 raft_consensus.cc:388] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:33:16.620657 20510 raft_consensus.cc:482] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:33:16.620718 20510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:16.621397 20510 raft_consensus.cc:504] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } }
I20260928 12:33:16.621502 20510 leader_election.cc:302] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afa29a2068c441fcadb29fcfc937a216; no voters:
I20260928 12:33:16.621698 20510 leader_election.cc:288] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:33:16.621994 20510 sys_catalog.cc:564] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:33:16.621789 20515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:33:16.622285 20515 raft_consensus.cc:686] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 1 LEADER]: Becoming Leader. State: Replica: afa29a2068c441fcadb29fcfc937a216, State: Running, Role: LEADER
I20260928 12:33:16.622416 20515 consensus_queue.cc:237] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } }
Sep 28 12:33:16 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:16.624017 20510 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 12:33:16.624284 20529 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afa29a2068c441fcadb29fcfc937a216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } } }
I20260928 12:33:16.624425 20529 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:16.626061 20515 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afa29a2068c441fcadb29fcfc937a216. Latest consensus state: current_term: 1 leader_uuid: "afa29a2068c441fcadb29fcfc937a216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } } }
I20260928 12:33:16.626247 20515 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:16.627523 20533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:33:16.628165 20533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:33:16.631105 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:16.615119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.8.13.62:33861)
0928 12:33:16.615322 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:16.615332 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:16.615394 (+ 62us) server_negotiation.cc:407] Connection header received
0928 12:33:16.617313 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:16.617322 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:16.617385 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:16.617528 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:16.619164 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:16.620318 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:16.621247 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:16.621765 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:16.621921 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:16.627081 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:16.627115 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:16.627119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:16.627203 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:16.629458 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:16.629896 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:16.629899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:16.629900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:16.629988 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:16.630180 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:16.630181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:16.630182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:16.630508 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:16.630655 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:16.630750 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 12:33:16.630889 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260928 12:33:16.634315 20533 catalog_manager.cc:1347] Generated new cluster ID: df36b3b8f42147e49b756bdab1d42752
I20260928 12:33:16.634372 20533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:33:16.642854 20533 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:33:16.643667 20533 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:33:16.650138 20533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216: Generated new TSK 0
I20260928 12:33:16.650408 20533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:33:17.397471 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33861
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:17.517972 20556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:17.518260 20556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:17.518296 20556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:17.520074 20556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:17.520145 20556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:17.520185 20556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:33:17.522569 20556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.13.20:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:17.522832 20556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:17.523129 20556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:17.529462 20561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:17.529783 20556 server_base.cc:1034] running on GCE node
W20260928 12:33:17.529556 20564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:17.529603 20562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:17.530117 20556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:17.530406 20556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:17.531594 20556 hybrid_clock.cc:648] HybridClock initialized: now 1790598797531572 us; error 36 us; skew 500 ppm
Sep 28 12:33:17 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598797, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:17.534858 20556 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:33:17.535416 20556 webserver.cc:460] Webserver started at http://127.8.13.1:46227/ using document root <none> and password file <none>
I20260928 12:33:17.535714 20556 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:17.535789 20556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:17.535962 20556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:33:17 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598797, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:33:17.550879 20556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "cdc9d60b96b646e58b1908239d0ede86"
format_stamp: "Formatted at 2026-09-28 12:33:17 on dist-test-slave-g31s"
server_key: "852ae25bcc7bb891697b3fc569046d7c"
server_key_iv: "e440e93202db96dcaf00114c5aac8277"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:17.551415 20556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "cdc9d60b96b646e58b1908239d0ede86"
format_stamp: "Formatted at 2026-09-28 12:33:17 on dist-test-slave-g31s"
server_key: "852ae25bcc7bb891697b3fc569046d7c"
server_key_iv: "e440e93202db96dcaf00114c5aac8277"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:17.553284 20556 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 12:33:17.571475 20572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:17.571718 20556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:17.571821 20556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "cdc9d60b96b646e58b1908239d0ede86"
format_stamp: "Formatted at 2026-09-28 12:33:17 on dist-test-slave-g31s"
server_key: "852ae25bcc7bb891697b3fc569046d7c"
server_key_iv: "e440e93202db96dcaf00114c5aac8277"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:17.571985 20556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:17.597970 20556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:17.600520 20556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:17.600737 20556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:17.601058 20556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:17.601477 20556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:17.601574 20556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:17.601671 20556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:17.601742 20556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:17.608279 20556 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:40025
I20260928 12:33:17.608717 20686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:40025 every 8 connection(s)
I20260928 12:33:17.608803 20556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:33:17 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598797, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:17.617888 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 20556
I20260928 12:33:17.618002 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:33:17.618176 8244 external_mini_cluster.cc:1428] Setting key af00c871e65192bb435115ef432e4756
I20260928 12:33:17.621485 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:17.609677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:41213 (local address 127.8.13.62:33861)
0928 12:33:17.609833 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:17.609836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:17.610772 (+ 936us) server_negotiation.cc:407] Connection header received
0928 12:33:17.610800 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:17.610801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:17.610813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:17.610866 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:17.611712 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:17.612532 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:17.614237 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:17.614514 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:17.614562 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:17.617264 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:17.617266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:17.617266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:17.617274 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:17.619737 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:17.620295 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:17.620297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:17.620297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:17.620366 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:17.620667 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:17.620668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:17.620668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:17.620812 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:17.620856 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:17.621285 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 12:33:17.621382 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260928 12:33:17.622305 20689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:17.609643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33861 (local address 127.8.13.1:41213)
0928 12:33:17.609926 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:17.609973 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:17.610601 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:17.610984 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:17.610988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:17.611235 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:17.611523 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:17.611580 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:17.612873 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:17.612874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:17.614127 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:17.614130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:17.614665 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:17.614665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:17.614799 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:17.615449 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:17.615459 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:17.617133 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:17.619842 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:17.619853 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:17.619856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:17.620201 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:17.620434 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:17.620435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:17.620435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:17.620588 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:17.620905 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:17.620912 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:17.621122 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 12:33:17.621869 (+ 747us) client_negotiation.cc:240] Negotiation successful
0928 12:33:17.622067 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20260928 12:33:17.624528 20687 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33861
I20260928 12:33:17.624639 20687 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:17.624847 20687 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
I20260928 12:33:17.625533 20474 ts_manager.cc:194] Registered new tserver with Master: cdc9d60b96b646e58b1908239d0ede86 (127.8.13.1:40025)
I20260928 12:33:17.626490 20474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:41213
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:33:17.672155 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33861
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:33:17 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598797, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 12:33:17.795316 20693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:17.795644 20693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:17.795718 20693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:17.797549 20693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:17.797676 20693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:17.797942 20693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:33:17.800402 20693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.13.20:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:17.800876 20693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:17.801188 20693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:17.807597 20701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:17.808259 20700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:17.808657 20703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:17.809242 20693 server_base.cc:1034] running on GCE node
I20260928 12:33:17.809476 20693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:17.809757 20693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:17.817103 20693 hybrid_clock.cc:648] HybridClock initialized: now 1790598797817075 us; error 40 us; skew 500 ppm
Sep 28 12:33:17 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598797, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:17.820384 20693 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:33:17.820916 20693 webserver.cc:460] Webserver started at http://127.8.13.2:40735/ using document root <none> and password file <none>
I20260928 12:33:17.821179 20693 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:17.821241 20693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:17.821376 20693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:33:17 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598797, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:33:17.837482 20693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "d312e397889b4e51b2a895d5223269cb"
format_stamp: "Formatted at 2026-09-28 12:33:17 on dist-test-slave-g31s"
server_key: "92ed19f2f71de20f242ccf5db9c79f34"
server_key_iv: "b48a90e3cad37dd8809a0db7590ce666"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:17.838006 20693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "d312e397889b4e51b2a895d5223269cb"
format_stamp: "Formatted at 2026-09-28 12:33:17 on dist-test-slave-g31s"
server_key: "92ed19f2f71de20f242ccf5db9c79f34"
server_key_iv: "b48a90e3cad37dd8809a0db7590ce666"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:17.840577 20693 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 12:33:17.858284 20710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:17.859076 20693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 12:33:17.859171 20693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "d312e397889b4e51b2a895d5223269cb"
format_stamp: "Formatted at 2026-09-28 12:33:17 on dist-test-slave-g31s"
server_key: "92ed19f2f71de20f242ccf5db9c79f34"
server_key_iv: "b48a90e3cad37dd8809a0db7590ce666"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:17.859265 20693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:17.877475 20693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:17.881174 20693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:17.881320 20693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:17.881522 20693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:17.881906 20693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:17.881954 20693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:17.881985 20693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:17.882009 20693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:17.897042 20693 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:41347
I20260928 12:33:17.897472 20693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:33:17.900238 20824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:41347 every 8 connection(s)
I20260928 12:33:17.906126 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 20693
I20260928 12:33:17.906226 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:33:17.906417 8244 external_mini_cluster.cc:1428] Setting key b8c733d8dd37c8250e06e57793edb51e
Sep 28 12:33:17 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598797, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:17.912035 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:17.901091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:57659 (local address 127.8.13.62:33861)
0928 12:33:17.901199 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:17.901202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:17.902084 (+ 882us) server_negotiation.cc:407] Connection header received
0928 12:33:17.902766 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:17.902768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:17.902780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:17.902808 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:17.903631 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:17.904434 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:17.905945 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:17.906239 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:17.906287 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:17.908810 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:17.908812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:17.908812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:17.908821 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:17.910476 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:17.910970 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:17.910971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:17.910972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:17.911029 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:17.911293 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:17.911294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:17.911295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:17.911425 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:17.911463 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:17.911834 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 12:33:17.911926 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 12:33:17.912528 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:17.900863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33861 (local address 127.8.13.2:57659)
0928 12:33:17.901920 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:17.901987 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:17.902650 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:17.903002 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:17.903006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:17.903250 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:17.903522 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:17.903527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:17.904549 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:17.904550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:17.905791 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:17.905833 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:17.906392 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:17.906393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:17.906532 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:17.907135 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:17.907145 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:17.908704 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:17.910573 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:17.910580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:17.910583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:17.910869 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:17.911098 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:17.911098 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:17.911099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:17.911229 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:17.911505 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:17.911510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:17.911682 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 12:33:17.912133 (+ 451us) client_negotiation.cc:240] Negotiation successful
0928 12:33:17.912299 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"thread_start_us":157,"threads_started":1}
I20260928 12:33:17.913146 20825 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33861
I20260928 12:33:17.913251 20825 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:17.913450 20825 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
I20260928 12:33:17.913842 20474 ts_manager.cc:194] Registered new tserver with Master: d312e397889b4e51b2a895d5223269cb (127.8.13.2:41347)
I20260928 12:33:17.914354 20474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:57659
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:33:17.949198 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33861
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:18.099020 20830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:18.099318 20830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:18.099356 20830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:18.101334 20830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:18.101414 20830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:18.101459 20830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:33:18.103856 20830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.13.20:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:18.104211 20830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:18.104570 20830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:18.115773 20836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:18.116029 20838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:18.116842 20835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:18.117092 20830 server_base.cc:1034] running on GCE node
I20260928 12:33:18.117367 20830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:18.117676 20830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:18.118856 20830 hybrid_clock.cc:648] HybridClock initialized: now 1790598798118835 us; error 33 us; skew 500 ppm
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:18.123612 20830 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:33:18.124243 20830 webserver.cc:460] Webserver started at http://127.8.13.3:44149/ using document root <none> and password file <none>
I20260928 12:33:18.124541 20830 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:18.124594 20830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:18.124718 20830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:33:18.145459 20830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "921c9227c4574defb7b48a2911d5b4a2"
format_stamp: "Formatted at 2026-09-28 12:33:18 on dist-test-slave-g31s"
server_key: "0a6ee966e928d849a538a9a4c75813e8"
server_key_iv: "d025312d49f4a363dd4cb04eb21d5077"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:18.146054 20830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "921c9227c4574defb7b48a2911d5b4a2"
format_stamp: "Formatted at 2026-09-28 12:33:18 on dist-test-slave-g31s"
server_key: "0a6ee966e928d849a538a9a4c75813e8"
server_key_iv: "d025312d49f4a363dd4cb04eb21d5077"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:18.148257 20830 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 12:33:18.163564 20845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:18.163789 20830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:18.163888 20830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "921c9227c4574defb7b48a2911d5b4a2"
format_stamp: "Formatted at 2026-09-28 12:33:18 on dist-test-slave-g31s"
server_key: "0a6ee966e928d849a538a9a4c75813e8"
server_key_iv: "d025312d49f4a363dd4cb04eb21d5077"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:18.163975 20830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:18.175112 20830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:18.177470 20830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:18.177668 20830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:18.178017 20830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:18.178436 20830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:18.178483 20830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:18.178514 20830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:18.178534 20830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:18.212858 20830 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:37585
I20260928 12:33:18.212927 20958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:37585 every 8 connection(s)
I20260928 12:33:18.213224 20830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:33:18.220854 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 20830
I20260928 12:33:18.220979 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:33:18.221175 8244 external_mini_cluster.cc:1428] Setting key 2044c34cc302f2638f12838eed7239c2
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:18.229446 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:18.214087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:52863 (local address 127.8.13.62:33861)
0928 12:33:18.214332 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:18.214334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:18.218229 (+ 3895us) server_negotiation.cc:407] Connection header received
0928 12:33:18.218920 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:18.218922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:18.218936 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:18.218964 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:18.219796 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.220614 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.222087 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.222410 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.222452 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:18.226240 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:18.226241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:18.226242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:18.226251 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:18.227827 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:18.228304 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:18.228306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:18.228306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:18.228364 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:18.228681 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:18.228682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:18.228682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:18.228815 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:18.228863 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:18.229228 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 12:33:18.229319 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260928 12:33:18.230420 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:18.214085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33861 (local address 127.8.13.3:52863)
0928 12:33:18.218072 (+ 3987us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:18.218125 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:18.218800 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:18.219156 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:18.219160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:18.219417 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:18.219691 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:18.219697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.220732 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.220733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:18.221990 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:18.221993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.222552 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.222553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:18.222688 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:18.223233 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:18.223251 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:18.226138 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:18.227919 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:18.227926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:18.227928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:18.228221 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:18.228477 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:18.228477 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:18.228478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:18.228611 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:18.228892 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:18.228899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:18.229073 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 12:33:18.230011 (+ 938us) client_negotiation.cc:240] Negotiation successful
0928 12:33:18.230183 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3942,"thread_start_us":93,"threads_started":1}
I20260928 12:33:18.231065 20959 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33861
I20260928 12:33:18.231204 20959 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:18.231436 20959 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
I20260928 12:33:18.231834 20474 ts_manager.cc:194] Registered new tserver with Master: 921c9227c4574defb7b48a2911d5b4a2 (127.8.13.3:37585)
I20260928 12:33:18.232336 20474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:52863
I20260928 12:33:18.233001 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:33:18.233531 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 20556
I20260928 12:33:18.240111 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 20693
I20260928 12:33:18.246804 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 20830
I20260928 12:33:18.250773 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 20442
I20260928 12:33:18.255247 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33861
--webserver_interface=127.8.13.62
--webserver_port=34169
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:33861
--ranger_config_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:35599/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.13.20:35599/kms with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:18.335518 20962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:18.335728 20962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:18.335758 20962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:18.336882 20962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:33:18.336926 20962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:18.336946 20962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:18.336961 20962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:33:18.336977 20962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:33:18.338621 20962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.13.20:35599/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:33861
--ranger_config_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=34169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:18.338853 20962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:18.339051 20962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:18.343689 20970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:18.343727 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:18.343950 20968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:18.343966 20962 server_base.cc:1034] running on GCE node
I20260928 12:33:18.344192 20962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:18.344458 20962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:18.345631 20962 hybrid_clock.cc:648] HybridClock initialized: now 1790598798345618 us; error 34 us; skew 500 ppm
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:18.348008 20962 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:33:18.348448 20962 webserver.cc:460] Webserver started at http://127.8.13.62:34169/ using document root <none> and password file <none>
I20260928 12:33:18.348654 20962 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:18.348702 20962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:33:18.401607 20962 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:18.402297 20978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:18.402498 20962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:33:18.402616 20962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "afa29a2068c441fcadb29fcfc937a216"
format_stamp: "Formatted at 2026-09-28 12:33:16 on dist-test-slave-g31s"
server_key: "d453e0a639e6546939a4ebe26aa1ee43"
server_key_iv: "d1d36b0b4cd887829174e94a865c9b0e"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:18.403118 20962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:18.434428 20962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:18.436496 20962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:18.436642 20962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:18.441497 20962 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:33861
I20260928 12:33:18.441525 21030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:33861 every 8 connection(s)
I20260928 12:33:18.442025 20962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:33:18.443261 21031 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:33:18.443890 21031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216: Bootstrap starting.
I20260928 12:33:18.450884 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 20962
I20260928 12:33:18.451211 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:40025
--local_ip_for_outbound_sockets=127.8.13.1
--tserver_master_addrs=127.8.13.62:33861
--webserver_port=46227
--webserver_interface=127.8.13.1
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.13.20:35599/kms with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:33:18.452272 21031 log.cc:824] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:18.454445 21031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:33:18.454716 21031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216: Bootstrap complete.
I20260928 12:33:18.457993 21031 raft_consensus.cc:348] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } }
I20260928 12:33:18.458331 21031 raft_consensus.cc:729] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa29a2068c441fcadb29fcfc937a216, State: Initialized, Role: FOLLOWER
I20260928 12:33:18.458557 21031 consensus_queue.cc:260] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } }
I20260928 12:33:18.458642 21031 raft_consensus.cc:388] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:33:18.458694 21031 raft_consensus.cc:482] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:33:18.458742 21031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:33:18.460093 21031 raft_consensus.cc:504] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } }
I20260928 12:33:18.460211 21031 leader_election.cc:302] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afa29a2068c441fcadb29fcfc937a216; no voters:
I20260928 12:33:18.460381 21031 leader_election.cc:288] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:33:18.460448 21035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:33:18.460603 21035 raft_consensus.cc:686] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [term 2 LEADER]: Becoming Leader. State: Replica: afa29a2068c441fcadb29fcfc937a216, State: Running, Role: LEADER
I20260928 12:33:18.460644 21031 sys_catalog.cc:564] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:33:18.460743 21035 consensus_queue.cc:237] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } }
I20260928 12:33:18.461027 21036 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "afa29a2068c441fcadb29fcfc937a216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } } }
I20260928 12:33:18.461119 21036 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:18.461025 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afa29a2068c441fcadb29fcfc937a216. Latest consensus state: current_term: 2 leader_uuid: "afa29a2068c441fcadb29fcfc937a216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29a2068c441fcadb29fcfc937a216" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33861 } } }
I20260928 12:33:18.461254 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216 [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:18.461308 21043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:33:18.461778 21043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:33:18.461954 21043 catalog_manager.cc:1259] Loaded cluster ID: df36b3b8f42147e49b756bdab1d42752
I20260928 12:33:18.461987 21043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:33:18.462158 21031 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 12:33:18.465406 21043 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:33:18.465986 21043 catalog_manager.cc:5879] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216: Loaded TSK: 0
I20260928 12:33:18.466238 21043 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:33:18.564329 21033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:18.564599 21033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:18.564632 21033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:18.566432 21033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:18.566565 21033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:18.566618 21033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:33:18.568819 21033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.13.20:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:40025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=46227
--tserver_master_addrs=127.8.13.62:33861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:18.569077 21033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:18.569329 21033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:18.575516 21068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:18.575536 21071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:18.575894 21069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:18.575987 21033 server_base.cc:1034] running on GCE node
I20260928 12:33:18.576229 21033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:18.576500 21033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:18.577679 21033 hybrid_clock.cc:648] HybridClock initialized: now 1790598798577660 us; error 35 us; skew 500 ppm
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:18.580880 21033 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:33:18.581418 21033 webserver.cc:460] Webserver started at http://127.8.13.1:46227/ using document root <none> and password file <none>
I20260928 12:33:18.581687 21033 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:18.581754 21033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:33:18.611840 21033 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.001s
I20260928 12:33:18.615346 21079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:18.615595 21033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 12:33:18.615710 21033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "cdc9d60b96b646e58b1908239d0ede86"
format_stamp: "Formatted at 2026-09-28 12:33:17 on dist-test-slave-g31s"
server_key: "852ae25bcc7bb891697b3fc569046d7c"
server_key_iv: "e440e93202db96dcaf00114c5aac8277"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:18.616212 21033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:18.627398 21033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:18.629526 21033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:18.629684 21033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:18.629982 21033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:18.630342 21033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:18.630389 21033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:18.630419 21033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:18.630435 21033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:18.637562 21033 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:40025
I20260928 12:33:18.637647 21192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:40025 every 8 connection(s)
I20260928 12:33:18.638056 21033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:18.647646 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 21033
I20260928 12:33:18.647953 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:41347
--local_ip_for_outbound_sockets=127.8.13.2
--tserver_master_addrs=127.8.13.62:33861
--webserver_port=40735
--webserver_interface=127.8.13.2
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.13.20:35599/kms with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:33:18.649530 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:18.638785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33861 (local address 127.8.13.1:54497)
0928 12:33:18.638949 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:18.638991 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:18.639493 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:18.639933 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:18.639937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:18.640158 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:18.640342 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:18.640346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.641479 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.641481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:18.642408 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:18.642411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.643030 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.643032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:18.643158 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:18.643664 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:18.643672 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:18.645178 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:18.647260 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:18.647266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:18.647268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:18.647533 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:18.647857 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:18.647858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:18.647858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:18.647973 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:18.648849 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:18.648855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:18.649003 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 12:33:18.649108 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 12:33:18.649266 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260928 12:33:18.650197 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:18.638916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:54497 (local address 127.8.13.62:33861)
0928 12:33:18.639107 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:18.639115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:18.639158 (+ 43us) server_negotiation.cc:407] Connection header received
0928 12:33:18.639708 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:18.639714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:18.639769 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:18.639887 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:18.640448 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.641385 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.642483 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.642922 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.643016 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:18.645299 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:18.645303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:18.645304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:18.645315 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:18.647170 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:18.647619 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:18.647622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:18.647622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:18.647796 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:18.648040 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:18.648062 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:18.648063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:18.648736 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:18.648819 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:18.649875 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0928 12:33:18.650006 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260928 12:33:18.650830 21193 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33861
I20260928 12:33:18.650925 21193 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:18.651101 21193 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
W20260928 12:33:18.651957 21193 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:33861 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 12:33:18.652029 21193 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:18.652096 21193 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
I20260928 12:33:18.652377 21193 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:18.652450 21193 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
W20260928 12:33:18.652711 21193 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:33:18.766243 21197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:18.766505 21197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:18.766541 21197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:18.768239 21197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:18.768301 21197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:18.768342 21197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:33:18.770569 21197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.13.20:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:41347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=40735
--tserver_master_addrs=127.8.13.62:33861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:18.770814 21197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:18.771096 21197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:18.778800 21203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:18.778987 21202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:18.778893 21205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:18.780311 21197 server_base.cc:1034] running on GCE node
I20260928 12:33:18.780542 21197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:18.780836 21197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:18.782009 21197 hybrid_clock.cc:648] HybridClock initialized: now 1790598798781994 us; error 30 us; skew 500 ppm
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:18.785043 21197 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:33:18.785622 21197 webserver.cc:460] Webserver started at http://127.8.13.2:40735/ using document root <none> and password file <none>
I20260928 12:33:18.785944 21197 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:18.786031 21197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:33:18.806660 21197 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 12:33:18.807716 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:18.807871 21197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:33:18.807963 21197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "d312e397889b4e51b2a895d5223269cb"
format_stamp: "Formatted at 2026-09-28 12:33:17 on dist-test-slave-g31s"
server_key: "92ed19f2f71de20f242ccf5db9c79f34"
server_key_iv: "b48a90e3cad37dd8809a0db7590ce666"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:18.808463 21197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:18.837924 21197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:18.839928 21197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:18.840045 21197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:18.840274 21197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:18.840629 21197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:18.840668 21197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:18.840698 21197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:18.840716 21197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:18.847007 21197 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:41347
I20260928 12:33:18.847463 21197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:33:18.847877 21326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:41347 every 8 connection(s)
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:18.855304 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 21197
I20260928 12:33:18.855609 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.20:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:37585
--local_ip_for_outbound_sockets=127.8.13.3
--tserver_master_addrs=127.8.13.62:33861
--webserver_port=44149
--webserver_interface=127.8.13.3
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.13.20:35599/kms with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:33:18.857895 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:18.848590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:42769 (local address 127.8.13.62:33861)
0928 12:33:18.848639 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:18.848641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:18.848943 (+ 302us) server_negotiation.cc:407] Connection header received
0928 12:33:18.849647 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:18.849649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:18.849662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:18.849733 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:18.850334 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.851092 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.852535 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.852731 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.852791 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:18.855079 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:18.855080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:18.855080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:18.855088 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:18.856416 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:18.856781 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:18.856782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:18.856783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:18.856845 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:18.857135 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:18.857136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:18.857137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:18.857284 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:18.857363 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:18.857684 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 12:33:18.857774 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:33:18.858059 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:18.848598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33861 (local address 127.8.13.2:42769)
0928 12:33:18.848792 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:18.848838 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:18.849536 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:18.849783 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:18.849786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:18.850021 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:18.850237 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:18.850242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.851204 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.851205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:18.852446 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:18.852449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:18.852873 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:18.852874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:18.853024 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:18.853559 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:18.853566 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:18.854992 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:18.856497 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:18.856503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:18.856505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:18.856713 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:18.856917 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:18.856918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:18.856918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:18.857056 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:18.857359 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:18.857364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:18.857523 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 12:33:18.857632 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 12:33:18.857792 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260928 12:33:18.858592 21327 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33861
I20260928 12:33:18.858688 21327 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:18.858863 21327 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
W20260928 12:33:18.859593 21327 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:33861 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 12:33:18.859668 21327 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:18.859722 21327 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
I20260928 12:33:18.860069 21327 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:18.860137 21327 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
W20260928 12:33:18.861083 21327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:33:18.952462 21330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:18.952737 21330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:18.952776 21330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:18.954632 21330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:18.954699 21330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:18.954741 21330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:33:18.956914 21330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.13.20:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:37585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=44149
--tserver_master_addrs=127.8.13.62:33861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:18.957182 21330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:18.957451 21330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:18.962616 21337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:18.962608 21339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:18.962982 21336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:18.963202 21330 server_base.cc:1034] running on GCE node
I20260928 12:33:18.963392 21330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:18.963620 21330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:18.964790 21330 hybrid_clock.cc:648] HybridClock initialized: now 1790598798964778 us; error 25 us; skew 500 ppm
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:18.967273 21330 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:33:18.967710 21330 webserver.cc:460] Webserver started at http://127.8.13.3:44149/ using document root <none> and password file <none>
I20260928 12:33:18.967922 21330 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:18.967985 21330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:33:18 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:33:18.995071 21330 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 12:33:18.996268 21347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:18.996450 21330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:33:18.996537 21330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "921c9227c4574defb7b48a2911d5b4a2"
format_stamp: "Formatted at 2026-09-28 12:33:18 on dist-test-slave-g31s"
server_key: "0a6ee966e928d849a538a9a4c75813e8"
server_key_iv: "d025312d49f4a363dd4cb04eb21d5077"
server_key_version: "kuduclusterkey@0"
I20260928 12:33:18.996887 21330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:19.015727 21330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:19.017962 21330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:19.018107 21330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:19.018329 21330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:19.018649 21330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:19.018684 21330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:19.018712 21330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:19.018730 21330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:19.025740 21330 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:37585
I20260928 12:33:19.028023 21460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:37585 every 8 connection(s)
I20260928 12:33:19.028559 21330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:33:19.033179 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 21330
Sep 28 12:33:19 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:19.039268 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:19.029619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:44897 (local address 127.8.13.62:33861)
0928 12:33:19.029670 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:19.029672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:19.029896 (+ 224us) server_negotiation.cc:407] Connection header received
0928 12:33:19.030477 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:19.030479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:19.030492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:19.030543 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:19.031059 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:19.031549 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:19.032963 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:19.033231 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:19.033279 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:19.035981 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:19.035982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:19.035982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:19.035990 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:19.037598 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:19.038113 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:19.038115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:19.038115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:19.038178 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:19.038477 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:19.038479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:19.038479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:19.038617 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:19.038685 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:19.039085 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 12:33:19.039171 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:33:19.039409 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:19.029610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33861 (local address 127.8.13.3:44897)
0928 12:33:19.029758 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:19.029820 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:19.030366 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:19.030607 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:19.030611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:19.030779 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:19.030962 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:19.030966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:19.031637 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:19.031638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:19.032857 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:19.032859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:19.033393 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:19.033395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:19.033706 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:19.034316 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:19.034326 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:19.035883 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:19.037686 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:19.037693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:19.037696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:19.038005 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:19.038231 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:19.038232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:19.038232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:19.038368 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:19.038716 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:19.038722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:19.038895 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 12:33:19.039009 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 12:33:19.039174 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260928 12:33:19.040040 21461 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33861
I20260928 12:33:19.040138 21461 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:19.040321 21461 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
W20260928 12:33:19.041034 21461 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:33861 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 12:33:19.041101 21461 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:19.041164 21461 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
I20260928 12:33:19.041435 21461 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:19.041503 21461 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
W20260928 12:33:19.041778 21461 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:33:19.050839 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:19.033609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.8.13.62:33861)
0928 12:33:19.034003 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:19.034004 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:19.034127 (+ 123us) server_negotiation.cc:407] Connection header received
0928 12:33:19.034952 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:19.034953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:19.034966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:19.035043 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:19.035553 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:19.036421 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:19.037395 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:19.037725 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:19.037822 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:19.042551 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:19.042553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:19.042554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:19.042561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:19.044216 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:19.044993 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:19.044995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:19.044995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:19.045061 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:19.050066 (+ 5005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:19.050067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:19.050067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:19.050207 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:19.050373 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:19.050634 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 12:33:19.050735 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":62,"threads_started":1}
Sep 28 12:33:19 dist-test-slave-g31s krb5kdc[19998](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598799, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:19.652918 21193 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:19.653085 21193 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
I20260928 12:33:19.653554 20995 ts_manager.cc:194] Registered new tserver with Master: cdc9d60b96b646e58b1908239d0ede86 (127.8.13.1:40025)
I20260928 12:33:19.654595 20995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:54497
I20260928 12:33:19.861254 21327 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:19.861440 21327 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
I20260928 12:33:19.861860 20995 ts_manager.cc:194] Registered new tserver with Master: d312e397889b4e51b2a895d5223269cb (127.8.13.2:41347)
I20260928 12:33:19.862422 20995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:42769
Sep 28 12:33:19 dist-test-slave-g31s krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598799, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for HTTP/127.8.13.20@KRBTEST.COM
I20260928 12:33:20.041962 21461 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:20.042140 21461 heartbeater.cc:511] Master 127.8.13.62:33861 requested a full tablet report, sending...
I20260928 12:33:20.042590 20995 ts_manager.cc:194] Registered new tserver with Master: 921c9227c4574defb7b48a2911d5b4a2 (127.8.13.3:37585)
I20260928 12:33:20.043148 20995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:44897
I20260928 12:33:20.044746 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:33:20.058751 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.050227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.8.13.62:33861)
0928 12:33:20.050380 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:20.050382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:20.050545 (+ 163us) server_negotiation.cc:407] Connection header received
0928 12:33:20.050621 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:20.050622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:20.050637 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:20.050727 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:20.051590 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.052369 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.053498 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.053791 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.053881 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.055532 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:20.055534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:20.055534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:20.055543 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:20.057371 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:20.057887 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:20.057888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:20.057889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:20.057960 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:20.058177 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:20.058178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:20.058178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:20.058315 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:20.058411 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:20.058558 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 12:33:20.058646 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260928 12:33:20.061100 20995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:33:20.063519 20995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:33:20.078020 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.067006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:41347 (local address 127.0.0.1:49008)
0928 12:33:20.067728 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:20.067738 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:20.067787 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:20.072793 (+ 5006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:20.072798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:20.072801 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:20.072927 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:20.072930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.074440 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.074441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:20.076281 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:20.076285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.077617 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.077618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:20.077726 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.077741 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 12:33:20.077842 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 12:33:20.077914 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":2,"thread_start_us":621,"threads_started":1}
I20260928 12:33:20.078198 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.067008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:40025 (local address 127.0.0.1:43648)
0928 12:33:20.073119 (+ 6111us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:20.073131 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:20.073176 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:20.073708 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:20.073709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:20.073712 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:20.073899 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:20.073902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.075304 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.075305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:20.076507 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:20.076510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.077774 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.077775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:20.077941 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.077947 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:33:20.077991 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:33:20.078041 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6065,"mutex_wait_us":2,"thread_start_us":197,"threads_started":1}
I20260928 12:33:20.078363 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.067006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:37585 (local address 127.0.0.1:46158)
0928 12:33:20.067484 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:20.067526 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:20.067784 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:20.074362 (+ 6578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:20.074363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:20.074366 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:20.074420 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:20.074421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.075733 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.075734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:20.076765 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:20.076767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.078043 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.078043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:20.078099 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.078101 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:33:20.078254 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 12:33:20.078290 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":337,"threads_started":1}
I20260928 12:33:20.078449 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.067462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49008 (local address 127.8.13.2:41347)
0928 12:33:20.072339 (+ 4877us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:20.072346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:20.072362 (+ 16us) server_negotiation.cc:407] Connection header received
0928 12:33:20.072405 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:20.072408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:20.072426 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:20.072520 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:20.073039 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.074321 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.076403 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.077435 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.077508 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.077578 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:20.078289 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 12:33:20.078343 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4818,"thread_start_us":47,"threads_started":1}
I20260928 12:33:20.078995 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.067230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.8.13.3:37585)
0928 12:33:20.074057 (+ 6827us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:20.074062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:20.074074 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:33:20.074109 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:20.074111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:20.074129 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:20.074211 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:20.074504 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.075638 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.076855 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.077888 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.077954 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.078010 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:20.078789 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 12:33:20.078836 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6785,"thread_start_us":54,"threads_started":1}
I20260928 12:33:20.079030 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.067307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.8.13.1:40025)
0928 12:33:20.073321 (+ 6014us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:20.073328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:20.073339 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:33:20.073374 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:20.073376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:20.073394 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:20.073480 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:20.073998 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.075203 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.076601 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.077564 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.077637 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.077698 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:20.078092 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 12:33:20.078805 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5970,"thread_start_us":48,"threads_started":1}
I20260928 12:33:20.079341 21259 tablet_service.cc:1467] Processing CreateTablet for tablet 09ac51dd6c6c4149bf60f1db73cfdf7e (DEFAULT_TABLE table=test-table [id=a6c271ea2e5e48989ce1f0b08007ff66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:20.079712 21259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ac51dd6c6c4149bf60f1db73cfdf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:20.079799 21395 tablet_service.cc:1467] Processing CreateTablet for tablet 09ac51dd6c6c4149bf60f1db73cfdf7e (DEFAULT_TABLE table=test-table [id=a6c271ea2e5e48989ce1f0b08007ff66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:20.079799 21109 tablet_service.cc:1467] Processing CreateTablet for tablet 09ac51dd6c6c4149bf60f1db73cfdf7e (DEFAULT_TABLE table=test-table [id=a6c271ea2e5e48989ce1f0b08007ff66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:20.080235 21395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ac51dd6c6c4149bf60f1db73cfdf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:20.080235 21109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ac51dd6c6c4149bf60f1db73cfdf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:20.085711 21496 tablet_bootstrap.cc:492] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2: Bootstrap starting.
I20260928 12:33:20.085693 21495 tablet_bootstrap.cc:492] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86: Bootstrap starting.
I20260928 12:33:20.087195 21495 tablet_bootstrap.cc:654] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:20.087195 21496 tablet_bootstrap.cc:654] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:20.088162 21495 log.cc:824] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:20.088158 21496 log.cc:824] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:20.088653 21494 tablet_bootstrap.cc:492] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb: Bootstrap starting.
I20260928 12:33:20.089648 21495 tablet_bootstrap.cc:492] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86: No bootstrap required, opened a new log
I20260928 12:33:20.089733 21495 ts_tablet_manager.cc:1397] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 12:33:20.090229 21496 tablet_bootstrap.cc:492] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2: No bootstrap required, opened a new log
I20260928 12:33:20.090355 21496 ts_tablet_manager.cc:1397] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 12:33:20.091042 21494 tablet_bootstrap.cc:654] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:20.092124 21494 log.cc:824] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:20.093302 21494 tablet_bootstrap.cc:492] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb: No bootstrap required, opened a new log
I20260928 12:33:20.093389 21494 ts_tablet_manager.cc:1397] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 12:33:20.095307 21495 raft_consensus.cc:348] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc9d60b96b646e58b1908239d0ede86" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40025 } } peers { permanent_uuid: "921c9227c4574defb7b48a2911d5b4a2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37585 } } peers { permanent_uuid: "d312e397889b4e51b2a895d5223269cb" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 41347 } }
I20260928 12:33:20.095539 21495 raft_consensus.cc:374] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:20.095588 21495 raft_consensus.cc:729] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc9d60b96b646e58b1908239d0ede86, State: Initialized, Role: FOLLOWER
I20260928 12:33:20.095754 21495 consensus_queue.cc:260] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc9d60b96b646e58b1908239d0ede86" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40025 } } peers { permanent_uuid: "921c9227c4574defb7b48a2911d5b4a2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37585 } } peers { permanent_uuid: "d312e397889b4e51b2a895d5223269cb" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 41347 } }
I20260928 12:33:20.095785 21496 raft_consensus.cc:348] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc9d60b96b646e58b1908239d0ede86" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40025 } } peers { permanent_uuid: "921c9227c4574defb7b48a2911d5b4a2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37585 } } peers { permanent_uuid: "d312e397889b4e51b2a895d5223269cb" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 41347 } }
I20260928 12:33:20.095932 21496 raft_consensus.cc:374] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:20.095980 21496 raft_consensus.cc:729] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 921c9227c4574defb7b48a2911d5b4a2, State: Initialized, Role: FOLLOWER
I20260928 12:33:20.096107 21496 consensus_queue.cc:260] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc9d60b96b646e58b1908239d0ede86" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40025 } } peers { permanent_uuid: "921c9227c4574defb7b48a2911d5b4a2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37585 } } peers { permanent_uuid: "d312e397889b4e51b2a895d5223269cb" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 41347 } }
I20260928 12:33:20.096205 21495 ts_tablet_manager.cc:1428] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260928 12:33:20.096340 21193 heartbeater.cc:503] Master 127.8.13.62:33861 was elected leader, sending a full tablet report...
I20260928 12:33:20.096436 21496 ts_tablet_manager.cc:1428] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 12:33:20.096544 21461 heartbeater.cc:503] Master 127.8.13.62:33861 was elected leader, sending a full tablet report...
I20260928 12:33:20.098290 21494 raft_consensus.cc:348] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc9d60b96b646e58b1908239d0ede86" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40025 } } peers { permanent_uuid: "921c9227c4574defb7b48a2911d5b4a2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37585 } } peers { permanent_uuid: "d312e397889b4e51b2a895d5223269cb" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 41347 } }
I20260928 12:33:20.098488 21494 raft_consensus.cc:374] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:20.098536 21494 raft_consensus.cc:729] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d312e397889b4e51b2a895d5223269cb, State: Initialized, Role: FOLLOWER
I20260928 12:33:20.098644 21494 consensus_queue.cc:260] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc9d60b96b646e58b1908239d0ede86" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40025 } } peers { permanent_uuid: "921c9227c4574defb7b48a2911d5b4a2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37585 } } peers { permanent_uuid: "d312e397889b4e51b2a895d5223269cb" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 41347 } }
I20260928 12:33:20.098980 21494 ts_tablet_manager.cc:1428] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260928 12:33:20.099040 21327 heartbeater.cc:503] Master 127.8.13.62:33861 was elected leader, sending a full tablet report...
W20260928 12:33:20.100189 21328 tablet.cc:2369] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:33:20.102037 21500 raft_consensus.cc:482] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:33:20.102181 21500 raft_consensus.cc:504] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc9d60b96b646e58b1908239d0ede86" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40025 } } peers { permanent_uuid: "921c9227c4574defb7b48a2911d5b4a2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37585 } } peers { permanent_uuid: "d312e397889b4e51b2a895d5223269cb" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 41347 } }
I20260928 12:33:20.102629 21500 leader_election.cc:288] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 921c9227c4574defb7b48a2911d5b4a2 (127.8.13.3:37585), d312e397889b4e51b2a895d5223269cb (127.8.13.2:41347)
I20260928 12:33:20.107328 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.103007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:40585 (local address 127.8.13.3:37585)
0928 12:33:20.103052 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:20.103054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:20.103119 (+ 65us) server_negotiation.cc:407] Connection header received
0928 12:33:20.103220 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:20.103221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:20.103234 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:20.103266 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:20.103665 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.104359 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.106075 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.106744 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.106781 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.106792 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:20.107210 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 12:33:20.107255 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:33:20.107748 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.102788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:37585 (local address 127.8.13.1:40585)
0928 12:33:20.103053 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:20.103064 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:20.103146 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:20.103334 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:20.103335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:20.103337 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:20.103565 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:20.103569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.104441 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.104441 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:20.105986 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:20.105989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.107004 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.107006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:20.107124 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.107128 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:33:20.107447 (+ 319us) client_negotiation.cc:240] Negotiation successful
0928 12:33:20.107492 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":155,"threads_started":1}
I20260928 12:33:20.108222 21415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ac51dd6c6c4149bf60f1db73cfdf7e" candidate_uuid: "cdc9d60b96b646e58b1908239d0ede86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921c9227c4574defb7b48a2911d5b4a2" is_pre_election: true
I20260928 12:33:20.108330 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.103413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:34897 (local address 127.8.13.2:41347)
0928 12:33:20.103462 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:20.103464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:20.103769 (+ 305us) server_negotiation.cc:407] Connection header received
0928 12:33:20.103858 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:20.103859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:20.103869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:20.103895 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:20.104331 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.104989 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.106774 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.107448 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.107487 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.107497 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:20.108225 (+ 728us) server_negotiation.cc:299] Negotiation successful
0928 12:33:20.108272 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:33:20.108387 21415 raft_consensus.cc:2454] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc9d60b96b646e58b1908239d0ede86 in term 0.
I20260928 12:33:20.108673 21082 leader_election.cc:302] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 921c9227c4574defb7b48a2911d5b4a2, cdc9d60b96b646e58b1908239d0ede86; no voters:
I20260928 12:33:20.108814 21500 raft_consensus.cc:2781] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:33:20.108860 21504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.103170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:41347 (local address 127.8.13.1:34897)
0928 12:33:20.103696 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:20.103705 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:20.103793 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:20.104042 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:20.104043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:20.104045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:20.104254 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:20.104256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.105083 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.105083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:20.106677 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:20.106680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.107765 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.107765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:20.108121 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.108126 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:33:20.108596 (+ 470us) client_negotiation.cc:240] Negotiation successful
0928 12:33:20.108641 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":157,"threads_started":1}
I20260928 12:33:20.108870 21500 raft_consensus.cc:482] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:33:20.109105 21500 raft_consensus.cc:3037] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:20.109400 21275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ac51dd6c6c4149bf60f1db73cfdf7e" candidate_uuid: "cdc9d60b96b646e58b1908239d0ede86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d312e397889b4e51b2a895d5223269cb" is_pre_election: true
I20260928 12:33:20.109612 21275 raft_consensus.cc:2454] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc9d60b96b646e58b1908239d0ede86 in term 0.
I20260928 12:33:20.110425 21500 raft_consensus.cc:504] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc9d60b96b646e58b1908239d0ede86" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40025 } } peers { permanent_uuid: "921c9227c4574defb7b48a2911d5b4a2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37585 } } peers { permanent_uuid: "d312e397889b4e51b2a895d5223269cb" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 41347 } }
I20260928 12:33:20.110548 21500 leader_election.cc:288] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [CANDIDATE]: Term 1 election: Requested vote from peers 921c9227c4574defb7b48a2911d5b4a2 (127.8.13.3:37585), d312e397889b4e51b2a895d5223269cb (127.8.13.2:41347)
I20260928 12:33:20.110770 21415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ac51dd6c6c4149bf60f1db73cfdf7e" candidate_uuid: "cdc9d60b96b646e58b1908239d0ede86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921c9227c4574defb7b48a2911d5b4a2"
I20260928 12:33:20.110829 21275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ac51dd6c6c4149bf60f1db73cfdf7e" candidate_uuid: "cdc9d60b96b646e58b1908239d0ede86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d312e397889b4e51b2a895d5223269cb"
I20260928 12:33:20.110849 21415 raft_consensus.cc:3037] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:20.110889 21275 raft_consensus.cc:3037] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:20.112151 21275 raft_consensus.cc:2454] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc9d60b96b646e58b1908239d0ede86 in term 1.
I20260928 12:33:20.112274 21415 raft_consensus.cc:2454] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc9d60b96b646e58b1908239d0ede86 in term 1.
I20260928 12:33:20.112447 21083 leader_election.cc:302] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdc9d60b96b646e58b1908239d0ede86, d312e397889b4e51b2a895d5223269cb; no voters:
I20260928 12:33:20.112582 21500 raft_consensus.cc:2781] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:33:20.112733 21500 raft_consensus.cc:686] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [term 1 LEADER]: Becoming Leader. State: Replica: cdc9d60b96b646e58b1908239d0ede86, State: Running, Role: LEADER
I20260928 12:33:20.112824 21500 consensus_queue.cc:237] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc9d60b96b646e58b1908239d0ede86" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40025 } } peers { permanent_uuid: "921c9227c4574defb7b48a2911d5b4a2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37585 } } peers { permanent_uuid: "d312e397889b4e51b2a895d5223269cb" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 41347 } }
I20260928 12:33:20.113586 20995 catalog_manager.cc:5495] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdc9d60b96b646e58b1908239d0ede86 (127.8.13.1). New cstate: current_term: 1 leader_uuid: "cdc9d60b96b646e58b1908239d0ede86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc9d60b96b646e58b1908239d0ede86" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "921c9227c4574defb7b48a2911d5b4a2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d312e397889b4e51b2a895d5223269cb" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 41347 } health_report { overall_health: UNKNOWN } } }
I20260928 12:33:20.119812 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.117417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.8.13.1:40025)
0928 12:33:20.117454 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:20.117456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:20.117463 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:33:20.117533 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:20.117534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:20.117551 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:20.117613 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:33:20.117735 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.118396 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.119124 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:20.119322 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:20.119376 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:20.119526 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:33:20.119628 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:33:20.119697 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:20.119722 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 12:33:20.119741 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:33:20.121943 21275 raft_consensus.cc:1261] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb [term 1 FOLLOWER]: Refusing update from remote peer cdc9d60b96b646e58b1908239d0ede86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:33:20.121943 21415 raft_consensus.cc:1261] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2 [term 1 FOLLOWER]: Refusing update from remote peer cdc9d60b96b646e58b1908239d0ede86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:33:20.122327 21500 consensus_queue.cc:1035] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "921c9227c4574defb7b48a2911d5b4a2" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:33:20.122530 21505 consensus_queue.cc:1035] T 09ac51dd6c6c4149bf60f1db73cfdf7e P cdc9d60b96b646e58b1908239d0ede86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d312e397889b4e51b2a895d5223269cb" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:33:20.125483 21509 mvcc.cc:205] Tried to move back new op lower bound from 7334292685296603136 to 7334292685262540800. Current Snapshot: MvccSnapshot[applied={T|T < 7334292685296603136}]
I20260928 12:33:20.125717 21508 mvcc.cc:205] Tried to move back new op lower bound from 7334292685296603136 to 7334292685262540800. Current Snapshot: MvccSnapshot[applied={T|T < 7334292685296603136}]
I20260928 12:33:20.127947 21510 mvcc.cc:205] Tried to move back new op lower bound from 7334292685296603136 to 7334292685262540800. Current Snapshot: MvccSnapshot[applied={T|T < 7334292685296603136}]
I20260928 12:33:20.129540 20994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:33:20.129628 20994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:33:20.130395 20994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afa29a2068c441fcadb29fcfc937a216: Sending DeleteTablet for 3 replicas of tablet 09ac51dd6c6c4149bf60f1db73cfdf7e
I20260928 12:33:20.131052 21109 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ac51dd6c6c4149bf60f1db73cfdf7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:20 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:43648
I20260928 12:33:20.131513 21395 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ac51dd6c6c4149bf60f1db73cfdf7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:20 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:46158
I20260928 12:33:20.131943 21259 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ac51dd6c6c4149bf60f1db73cfdf7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:20 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:49008
I20260928 12:33:20.132548 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 21033
I20260928 12:33:20.137729 21519 tablet_replica.cc:330] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2: stopping tablet replica
I20260928 12:33:20.137939 21520 tablet_replica.cc:330] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb: stopping tablet replica
I20260928 12:33:20.138077 21520 raft_consensus.cc:2229] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:33:20.138221 21520 raft_consensus.cc:2258] T 09ac51dd6c6c4149bf60f1db73cfdf7e P d312e397889b4e51b2a895d5223269cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:33:20.138427 21519 raft_consensus.cc:2229] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:33:20.138562 21519 raft_consensus.cc:2258] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 12:33:20.138684 20980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:40025 (error 108)
I20260928 12:33:20.139420 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 21197
I20260928 12:33:20.140937 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.140811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:40025 (local address 127.0.0.1:43664)
0928 12:33:20.140856 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:20.140898 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:40025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:33:20.145398 20980 catalog_manager.cc:4584] TS cdc9d60b96b646e58b1908239d0ede86 (127.8.13.1:40025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09ac51dd6c6c4149bf60f1db73cfdf7e: Network error: Client connection negotiation failed: client connection to 127.8.13.1:40025: connect: Connection refused (error 111)
I20260928 12:33:20.145941 21519 ts_tablet_manager.cc:1905] T 09ac51dd6c6c4149bf60f1db73cfdf7e P 921c9227c4574defb7b48a2911d5b4a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:33:20.145970 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 21330
I20260928 12:33:20.149685 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.149478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:41347 (local address 127.0.0.1:49010)
0928 12:33:20.149591 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:20.149638 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.2:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 12:33:20.153504 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:20.153364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:37585 (local address 127.0.0.1:46160)
0928 12:33:20.153428 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:20.153455 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.3:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:33:20.153509 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 20962
I20260928 12:33:20.159230 8244 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 12:33:20.493956 8244 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 12:33:20.494060 8244 mini_ranger.cc:66] Stopping Ranger...
I20260928 12:33:20.828333 8244 mini_ranger.cc:68] Stopped Ranger
2026-09-28 12:33:20.828 UTC [20028] LOG: received smart shutdown request
2026-09-28 12:33:20.832 UTC [20028] LOG: background worker "logical replication launcher" (PID 20037) exited with exit code 1
2026-09-28 12:33:20.832 UTC [20032] LOG: shutting down
2026-09-28 12:33:20.999 UTC [20028] LOG: database system is shut down
2026-09-28T12:33:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48012 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:33:21 dist-test-slave-g31s krb5kdc[21530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:33:21 dist-test-slave-g31s krb5kdc[21530](info): set up 2 sockets
Sep 28 12:33:21 dist-test-slave-g31s krb5kdc[21530](info): commencing operation
krb5kdc: starting...
W20260928 12:33:22.045650 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:33:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:33:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:33:22.162981 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:35555
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:35555,127.8.13.61:42537,127.8.13.60:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:22.246690 21543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:22.246920 21543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:22.246953 21543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:22.248243 21543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:33:22.248296 21543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:22.248310 21543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:22.248322 21543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:33:22.248346 21543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:33:22.250093 21543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:35555,127.8.13.61:42537,127.8.13.60:33779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:35555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:22.250353 21543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:22.250578 21543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:22.255447 21551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:22.255440 21549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:22.255654 21548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:22.255573 21543 server_base.cc:1034] running on GCE node
I20260928 12:33:22.255901 21543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:22.256197 21543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:22.257344 21543 hybrid_clock.cc:648] HybridClock initialized: now 1790598802257330 us; error 29 us; skew 500 ppm
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:22.259771 21543 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:33:22.260178 21543 webserver.cc:460] Webserver started at http://127.8.13.62:34665/ using document root <none> and password file <none>
I20260928 12:33:22.260355 21543 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:22.260394 21543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:22.260499 21543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:22.261564 21543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "333e637223c04f0a837a8cf5bdf579ca"
format_stamp: "Formatted at 2026-09-28 12:33:22 on dist-test-slave-g31s"
server_key: "9e659fe3de72a2fbd5f16b1dd49d75cf"
server_key_iv: "54ccc915fd5de38e08a70a31791c5632"
server_key_version: "encryptionkey@0"
I20260928 12:33:22.261932 21543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "333e637223c04f0a837a8cf5bdf579ca"
format_stamp: "Formatted at 2026-09-28 12:33:22 on dist-test-slave-g31s"
server_key: "9e659fe3de72a2fbd5f16b1dd49d75cf"
server_key_iv: "54ccc915fd5de38e08a70a31791c5632"
server_key_version: "encryptionkey@0"
I20260928 12:33:22.263200 21543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:33:22.263919 21558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:22.264088 21543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:33:22.264181 21543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "333e637223c04f0a837a8cf5bdf579ca"
format_stamp: "Formatted at 2026-09-28 12:33:22 on dist-test-slave-g31s"
server_key: "9e659fe3de72a2fbd5f16b1dd49d75cf"
server_key_iv: "54ccc915fd5de38e08a70a31791c5632"
server_key_version: "encryptionkey@0"
I20260928 12:33:22.264254 21543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:22.273465 21543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:22.275383 21543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:22.275512 21543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:22.279361 21543 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:35555
I20260928 12:33:22.279399 21610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:35555 every 8 connection(s)
I20260928 12:33:22.279735 21543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:33:22.280613 21611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:22.282186 21611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:33:22.287869 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 21543
I20260928 12:33:22.287976 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:22.288182 8244 external_mini_cluster.cc:1428] Setting key b44fb5c9f45888d1ffdb4137feb75fe5
I20260928 12:33:22.291826 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.282892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.8.13.62:35555)
0928 12:33:22.283073 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.283078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.283138 (+ 60us) server_negotiation.cc:407] Connection header received
0928 12:33:22.283689 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.283691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.283707 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.283785 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.284328 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.285130 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.286020 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.286326 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.286421 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.288588 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.288596 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.288596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.288603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.290296 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.290710 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.290712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.290712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.290792 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.291020 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.291021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.291021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.291233 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.291293 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.291503 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.291646 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260928 12:33:22.291848 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.282846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:35555 (local address 127.0.0.1:35866)
0928 12:33:22.283014 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.283057 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:22.283536 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:22.283797 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:22.283800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:22.284072 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:22.284228 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.284233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.285223 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.285224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.285943 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.285945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.286400 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.286401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.286465 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.287012 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:22.287020 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:22.288498 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:22.290381 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.290388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.290390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.290620 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.290853 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.290854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.290854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.290963 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.291291 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:22.291297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:22.291376 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 12:33:22.291504 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 12:33:22.291740 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":74,"thread_start_us":64,"threads_started":1}
I20260928 12:33:22.292790 21611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:33:22.293183 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.293047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:42537 (local address 127.0.0.1:41922)
0928 12:33:22.293103 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.293157 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 12:33:22.293360 21559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.61:42537: connect: Connection refused (error 111)
I20260928 12:33:22.293826 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.293702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:42537 (local address 127.0.0.1:41938)
0928 12:33:22.293750 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.293780 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:33:22.293915 21611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.61:42537: Network error: Client connection negotiation failed: client connection to 127.8.13.61:42537: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.8.13.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab.
Entry for principal kudu/127.8.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab.
I20260928 12:33:22.313658 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.61:42537
--webserver_interface=127.8.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab
--principal=kudu/127.8.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:35555,127.8.13.61:42537,127.8.13.60:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:33:22.343165 21611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } attempt: 1
I20260928 12:33:22.343729 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.343561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:42537 (local address 127.0.0.1:41954)
0928 12:33:22.343637 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.343691 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 12:33:22.344074 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.343978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:42537 (local address 127.0.0.1:41970)
0928 12:33:22.344024 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.344053 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:33:22.344159 21611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.61:42537: Network error: Client connection negotiation failed: client connection to 127.8.13.61:42537: connect: Connection refused (error 111)
W20260928 12:33:22.394919 21618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:22.395179 21618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:22.395205 21618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:22.396440 21618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:33:22.396484 21618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:22.396497 21618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:22.396508 21618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:33:22.396520 21618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:33:22.398167 21618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:35555,127.8.13.61:42537,127.8.13.60:33779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab
--principal=kudu/127.8.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.61:42537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:22.398427 21618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:22.398644 21618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:22.403456 21624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:22.403450 21626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:22.403584 21623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:22.403748 21618 server_base.cc:1034] running on GCE node
I20260928 12:33:22.404000 21618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:22.404341 21618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:22.405514 21618 hybrid_clock.cc:648] HybridClock initialized: now 1790598802405498 us; error 31 us; skew 500 ppm
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:22.408341 21618 init.cc:377] Logged in from keytab as kudu/127.8.13.61@KRBTEST.COM (short username kudu)
I20260928 12:33:22.408872 21618 webserver.cc:460] Webserver started at http://127.8.13.61:38587/ using document root <none> and password file <none>
I20260928 12:33:22.409096 21618 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:22.409143 21618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:22.409272 21618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:22.410425 21618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/data/instance:
uuid: "c79ecfc859f042c5951252d37045a9b0"
format_stamp: "Formatted at 2026-09-28 12:33:22 on dist-test-slave-g31s"
server_key: "0093d900a080c655259bb925474c6f7c"
server_key_iv: "b92b43588fafc62045ef951afdcd1fde"
server_key_version: "encryptionkey@0"
I20260928 12:33:22.410768 21618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/wal/instance:
uuid: "c79ecfc859f042c5951252d37045a9b0"
format_stamp: "Formatted at 2026-09-28 12:33:22 on dist-test-slave-g31s"
server_key: "0093d900a080c655259bb925474c6f7c"
server_key_iv: "b92b43588fafc62045ef951afdcd1fde"
server_key_version: "encryptionkey@0"
I20260928 12:33:22.412177 21618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:33:22.412293 21611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } attempt: 2
I20260928 12:33:22.412662 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.412549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:42537 (local address 127.0.0.1:41974)
0928 12:33:22.412602 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.412633 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 12:33:22.413012 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.412921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:42537 (local address 127.0.0.1:41982)
0928 12:33:22.412964 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.412994 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:33:22.413065 21633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260928 12:33:22.413120 21611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.61:42537: Network error: Client connection negotiation failed: client connection to 127.8.13.61:42537: connect: Connection refused (error 111)
I20260928 12:33:22.413232 21618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:33:22.413311 21618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/wal
uuid: "c79ecfc859f042c5951252d37045a9b0"
format_stamp: "Formatted at 2026-09-28 12:33:22 on dist-test-slave-g31s"
server_key: "0093d900a080c655259bb925474c6f7c"
server_key_iv: "b92b43588fafc62045ef951afdcd1fde"
server_key_version: "encryptionkey@0"
I20260928 12:33:22.413389 21618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:22.440038 21618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:22.442180 21618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:22.442302 21618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:22.446233 21618 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.61:42537
I20260928 12:33:22.446267 21685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.61:42537 every 8 connection(s)
I20260928 12:33:22.446631 21618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
I20260928 12:33:22.447576 21686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:22.448843 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 21618
I20260928 12:33:22.448968 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-1/wal/instance
I20260928 12:33:22.449155 8244 external_mini_cluster.cc:1428] Setting key 2ab9f32a8aaaec7f0fb1930f6d664556
I20260928 12:33:22.449570 21686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.60@KRBTEST.COM; defaulting to no policy
I20260928 12:33:22.459553 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.450460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.8.13.62:35555)
0928 12:33:22.450512 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.450514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.450816 (+ 302us) server_negotiation.cc:407] Connection header received
0928 12:33:22.451294 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.451296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.451310 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.451391 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.452047 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.452786 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.453877 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.454210 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.454289 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.456707 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.456709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.456709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.456719 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.458198 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.458616 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.458617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.458617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.458667 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.458925 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.458925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.458926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.459070 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.459121 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.459374 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.459444 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.60@KRBTEST.COM" created.
I20260928 12:33:22.459738 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.450467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:35555 (local address 127.0.0.1:35882)
0928 12:33:22.450670 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.450713 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:22.451198 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:22.451465 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:22.451469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:22.451737 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:22.451953 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.451957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.452918 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.452919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.453761 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.453763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.454293 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.454294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.454413 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.454930 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:22.454939 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:22.456595 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:22.458295 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.458305 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.458308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.458554 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.458717 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.458718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.458718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.458829 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.459126 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:22.459131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:22.459266 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 12:33:22.459355 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:33:22.459504 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":110,"threads_started":1}
I20260928 12:33:22.460440 21686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
I20260928 12:33:22.468619 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.460711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:42537 (local address 127.0.0.1:41994)
0928 12:33:22.460758 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.460767 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:22.460799 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:22.461028 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:22.461028 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:22.461176 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:22.461244 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.461246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.462489 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.462490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.463018 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.463020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.463641 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.463642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.463703 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.464108 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:22.464112 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:22.465478 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:22.467551 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.467552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.467553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.467719 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.467906 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.467906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.467907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.467956 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.468235 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:22.468236 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:22.468282 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:33:22.468351 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 12:33:22.468522 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":18}
I20260928 12:33:22.468688 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.460758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.8.13.61:42537)
0928 12:33:22.460904 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.460908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.460923 (+ 15us) server_negotiation.cc:407] Connection header received
0928 12:33:22.460941 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.460942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.460953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.461012 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.461313 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.462393 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.463095 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.463552 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.463628 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.465568 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.465579 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.465580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.465588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.467471 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.467794 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.467797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.467798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.467849 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.467999 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.467999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.467999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.468145 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.468204 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.468400 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.468572 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260928 12:33:22.469339 21686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:33:22.469734 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.469587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50888)
0928 12:33:22.469657 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.469710 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 12:33:22.469911 21637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
I20260928 12:33:22.470341 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.470237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50892)
0928 12:33:22.470284 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.470316 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 12:33:22.470489 21686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:33779: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab.
Entry for principal kudu/127.8.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab.
I20260928 12:33:22.477311 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.60:33779
--webserver_interface=127.8.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab
--principal=kudu/127.8.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:35555,127.8.13.61:42537,127.8.13.60:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:33:22.504290 21611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } attempt: 3
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
I20260928 12:33:22.511848 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.504693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:42537 (local address 127.0.0.1:42004)
0928 12:33:22.504777 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.504791 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:22.504856 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:22.504999 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:22.505000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:22.505186 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:22.505262 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.505265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.506011 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.506012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.506805 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.506809 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.507203 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.507204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.507282 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.507743 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:22.507748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:22.509170 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:22.510841 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.510843 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.510843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.511014 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.511245 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.511246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.511247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.511310 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.511583 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:22.511584 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:22.511628 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 12:33:22.511685 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:33:22.511773 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 12:33:22.511919 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.504773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.8.13.61:42537)
0928 12:33:22.504833 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.504835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.504845 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:33:22.504919 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.504920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.504930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.504977 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.505363 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.505931 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.506900 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.507128 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.507185 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.509257 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.509259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.509259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.509268 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.510739 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.511089 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.511090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.511091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.511152 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.511376 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.511377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.511377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.511510 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.511577 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.511727 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.511825 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:33:22.512326 21611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:33:22.512669 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.512571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50906)
0928 12:33:22.512617 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.512645 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 12:33:22.512948 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.512886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50916)
0928 12:33:22.512912 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.512934 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 12:33:22.513028 21611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:33779: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
I20260928 12:33:22.519722 21686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } attempt: 1
I20260928 12:33:22.520227 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.520049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50922)
0928 12:33:22.520142 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.520191 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 12:33:22.520566 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.520484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50934)
0928 12:33:22.520522 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.520546 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 12:33:22.520648 21686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:33779: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
I20260928 12:33:22.544214 21611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } attempt: 1
I20260928 12:33:22.544725 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.544591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50946)
0928 12:33:22.544653 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.544695 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 12:33:22.545087 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.544986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50962)
0928 12:33:22.545029 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.545069 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:33:22.545179 21611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:33779: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
W20260928 12:33:22.561184 21693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:22.561420 21693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:22.561444 21693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:22.562718 21693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:33:22.562783 21693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:22.562795 21693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:22.562808 21693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:33:22.562819 21693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:33:22.564481 21693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:35555,127.8.13.61:42537,127.8.13.60:33779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab
--principal=kudu/127.8.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.60:33779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:22.564705 21693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:22.564905 21693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:22.569926 21698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:22.569931 21701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:22.569957 21693 server_base.cc:1034] running on GCE node
W20260928 12:33:22.569926 21699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:22.570423 21693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:22.570757 21693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:22.571906 21693 hybrid_clock.cc:648] HybridClock initialized: now 1790598802571891 us; error 31 us; skew 500 ppm
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:22.574440 21693 init.cc:377] Logged in from keytab as kudu/127.8.13.60@KRBTEST.COM (short username kudu)
I20260928 12:33:22.574905 21693 webserver.cc:460] Webserver started at http://127.8.13.60:40437/ using document root <none> and password file <none>
I20260928 12:33:22.575135 21693 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:22.575186 21693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:22.575315 21693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:22.576388 21693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/data/instance:
uuid: "e0c54291811041338389e3fc125c1f7a"
format_stamp: "Formatted at 2026-09-28 12:33:22 on dist-test-slave-g31s"
server_key: "f7def18b8be845ebab7ac861bd10e45d"
server_key_iv: "ae939f319c366edec524085ee2670531"
server_key_version: "encryptionkey@0"
I20260928 12:33:22.576730 21693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/wal/instance:
uuid: "e0c54291811041338389e3fc125c1f7a"
format_stamp: "Formatted at 2026-09-28 12:33:22 on dist-test-slave-g31s"
server_key: "f7def18b8be845ebab7ac861bd10e45d"
server_key_iv: "ae939f319c366edec524085ee2670531"
server_key_version: "encryptionkey@0"
I20260928 12:33:22.578120 21693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:33:22.578877 21708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:22.579051 21693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:33:22.579140 21693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/wal
uuid: "e0c54291811041338389e3fc125c1f7a"
format_stamp: "Formatted at 2026-09-28 12:33:22 on dist-test-slave-g31s"
server_key: "f7def18b8be845ebab7ac861bd10e45d"
server_key_iv: "ae939f319c366edec524085ee2670531"
server_key_version: "encryptionkey@0"
I20260928 12:33:22.579205 21693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:22.588798 21686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } attempt: 2
I20260928 12:33:22.589349 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.589204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50964)
0928 12:33:22.589261 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.589305 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 12:33:22.589660 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.589590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50970)
0928 12:33:22.589623 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.589645 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 12:33:22.589761 21686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:33779: Network error: Client connection negotiation failed: client connection to 127.8.13.60:33779: connect: Connection refused (error 111)
I20260928 12:33:22.594404 21693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:22.602550 21693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:22.602754 21693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:22.606806 21693 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.60:33779
I20260928 12:33:22.606840 21760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.60:33779 every 8 connection(s)
I20260928 12:33:22.607196 21693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
I20260928 12:33:22.608071 21761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:22.609763 21761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:33:22.612407 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 21693
I20260928 12:33:22.612517 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598640018661-8244-0/minicluster-data/master-2/wal/instance
I20260928 12:33:22.612703 8244 external_mini_cluster.cc:1428] Setting key ddf4dba1a1c26fc18150e24b973ace77
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
I20260928 12:33:22.619261 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.610514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.8.13.62:35555)
0928 12:33:22.610576 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.610578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.610763 (+ 185us) server_negotiation.cc:407] Connection header received
0928 12:33:22.611263 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.611265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.611277 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.611330 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.611988 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.612602 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.614026 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.614214 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.614277 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.616557 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.616559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.616559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.616565 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.617976 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.618370 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.618372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.618372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.618432 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.618652 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.618654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.618654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.618771 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.618829 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.619088 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.619170 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:33:22.619414 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.610459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:35555 (local address 127.0.0.1:35894)
0928 12:33:22.610640 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.610679 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:22.611156 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:22.611414 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:22.611419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:22.611681 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:22.611892 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.611897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.612713 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.612714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.613932 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.613935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.614292 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.614293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.614435 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.614963 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:22.614972 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:22.616474 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:22.618047 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.618053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.618054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.618296 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.618497 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.618497 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.618497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.618588 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.618828 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:22.618835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:22.618951 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 12:33:22.619055 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 12:33:22.619204 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260928 12:33:22.619933 21761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:33:22.620332 21611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } attempt: 2
I20260928 12:33:22.621711 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.613069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.8.13.60:33779)
0928 12:33:22.613227 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.613230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.613245 (+ 15us) server_negotiation.cc:407] Connection header received
0928 12:33:22.613297 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.613298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.613313 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.613374 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.614102 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.614867 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.615820 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.616145 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.616226 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.618627 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.618635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.618635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.618642 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.620485 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.620844 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.620847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.620847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.620917 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.621130 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.621131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.621131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.621288 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.621456 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.621535 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.621610 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:22.627355 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.620166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:42537 (local address 127.0.0.1:42012)
0928 12:33:22.620205 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.620214 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:22.620247 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:22.620393 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:22.620394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:22.620557 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:22.620596 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.620598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.621449 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.621450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.622118 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.622120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.622448 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.622449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.622505 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.622905 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:22.622909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:22.624563 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:22.626268 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.626269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.626270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.626532 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.626739 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.626740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.626740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.626802 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.627063 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:22.627064 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:22.627108 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 12:33:22.627175 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 12:33:22.627259 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 12:33:22.627373 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.620236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42012 (local address 127.8.13.61:42537)
0928 12:33:22.620301 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.620303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.620310 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:33:22.620332 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.620333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.620342 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.620385 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.620680 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.621367 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.622195 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.622389 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.622443 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.624641 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.624642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.624643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.624650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.626180 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.626608 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.626609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.626610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.626668 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.626864 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.626864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.626864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.626997 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.627060 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.627213 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.627285 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:33:22.627992 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.620593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50982)
0928 12:33:22.620636 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.620645 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:22.620676 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:22.620873 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:22.620874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:22.621014 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:22.621053 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.621055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.621825 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.621826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.622329 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.622331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.623022 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.623023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.623083 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.623429 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:22.623434 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:22.624617 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:22.627017 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.627018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.627019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.627204 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.627416 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.627417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.627418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.627487 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.627706 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:22.627707 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:22.627747 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 12:33:22.627817 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:33:22.627901 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:33:22.627969 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.620634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.8.13.60:33779)
0928 12:33:22.620777 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.620778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.620784 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:33:22.620798 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.620799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.620807 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.620863 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.621120 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.621712 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.622576 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.622946 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.623003 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.624692 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.624693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.624694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.624698 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.626931 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.627284 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.627285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.627285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.627344 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.627557 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.627558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.627558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.627654 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.627699 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.627833 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.627891 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260928 12:33:22.628579 21761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:33:22.629781 21611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca: Bootstrap starting.
I20260928 12:33:22.630703 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.623074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.8.13.62:35555)
0928 12:33:22.623110 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.623112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.623119 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:33:22.623168 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.623169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.623177 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.623222 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.623747 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.624221 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.625079 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.625349 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.625410 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.628111 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.628112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.628112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.628118 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.629567 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.629890 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.629891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.629891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.629941 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.630111 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.630112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.630112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.630228 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.630354 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.630547 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.630618 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:33:22.630705 21611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:22.631106 21611 log.cc:824] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:22.631919 21611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca: No bootstrap required, opened a new log
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
I20260928 12:33:22.633982 21611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } }
I20260928 12:33:22.634312 21611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:22.634377 21611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333e637223c04f0a837a8cf5bdf579ca, State: Initialized, Role: FOLLOWER
I20260928 12:33:22.634532 21611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } }
I20260928 12:33:22.634833 21768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } } }
I20260928 12:33:22.634918 21611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:33:22.634980 21768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
I20260928 12:33:22.635794 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.628810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:50992)
0928 12:33:22.629004 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.629014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:22.629061 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:22.629179 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:22.629180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:22.629305 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:22.629362 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.629364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.630185 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.630186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.630852 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.630854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.631264 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.631265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.631308 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.631651 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:22.631655 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:22.632939 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:22.634632 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.634633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.634634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.634839 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.635023 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.635023 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.635024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.635067 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.635436 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:22.635437 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:22.635483 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:33:22.635577 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 12:33:22.635680 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260928 12:33:22.635874 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.628853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.8.13.60:33779)
0928 12:33:22.628947 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.628949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.629064 (+ 115us) server_negotiation.cc:407] Connection header received
0928 12:33:22.629116 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.629116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.629125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.629178 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.629438 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.630118 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.630929 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.631210 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.631268 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.633026 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.633028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.633028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.633035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.634549 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.634923 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.634924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.634924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.634965 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.635141 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.635142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.635143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.635345 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.635434 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.635615 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.635743 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":75}
I20260928 12:33:22.638748 21761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a: Bootstrap starting.
I20260928 12:33:22.639082 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.631532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.8.13.61:42537)
0928 12:33:22.631565 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.631567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.631573 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:33:22.631592 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.631592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.631601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.631642 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.632180 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.632626 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.633379 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.633727 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.633770 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.636156 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.636158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.636158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.636165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.637764 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.638167 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.638168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.638168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.638224 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.638412 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.638413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.638414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.638558 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.638700 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.638914 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.638998 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:33:22.639514 21761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:22.639730 21761 log.cc:824] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:22.640430 21761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a: No bootstrap required, opened a new log
I20260928 12:33:22.638185 21543 master_runner.cc:186] Error getting master registration for 127.8.13.61:42537: OK, instance_id { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" instance_seqno: 1790598802445084 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:33:22.641031 21543 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 12:33:22.642314 21779 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 12:33:22.642227 21761 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } }
W20260928 12:33:22.642369 21779 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:33:22.642376 21761 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:22.642409 21761 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c54291811041338389e3fc125c1f7a, State: Initialized, Role: FOLLOWER
I20260928 12:33:22.642539 21761 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } }
I20260928 12:33:22.642871 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:33:22.642880 21761 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [sys.catalog]: configured and running, proceeding with master startup.
W20260928 12:33:22.647928 21799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 12:33:22.642841 21782 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } } }
I20260928 12:33:22.650406 21782 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:33:22.650487 21799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:33:22.649358 21693 master_runner.cc:186] Error getting master registration for 127.8.13.61:42537: OK, instance_id { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" instance_seqno: 1790598802445084 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:33:22.651863 21693 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:33:22.654212 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.644339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.8.13.61:42537)
0928 12:33:22.644379 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.644381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.644621 (+ 240us) server_negotiation.cc:407] Connection header received
0928 12:33:22.644682 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.644683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.644694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.644742 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.645522 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.647835 (+ 2313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.649085 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.649405 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.649450 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.651282 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.651284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.651284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.651292 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.653013 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.653355 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.653356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.653357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.653410 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.653604 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.653605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.653605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.653730 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.653830 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.654036 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.654120 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:33:22.654927 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.644333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.8.13.62:35555)
0928 12:33:22.644391 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.644393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.644757 (+ 364us) server_negotiation.cc:407] Connection header received
0928 12:33:22.644840 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.644841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.644851 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.644878 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.645515 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.646192 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.647414 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.647726 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.647763 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.650551 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.650553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.650554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.650563 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.652423 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.654004 (+ 1581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.654005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.654006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.654084 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.654318 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.654320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.654320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.654453 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.654538 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.654744 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.654830 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:33:22.659677 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.646908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.8.13.60:33779)
0928 12:33:22.648352 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.648354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.648457 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:33:22.648573 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.648574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.648586 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.648615 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.652942 (+ 4327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.653847 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.655125 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.655398 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.655434 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.656720 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.656721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.656722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.656729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.658300 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.658707 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.658708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.658709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.658772 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.659011 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.659012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.659012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.659163 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.659250 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.659478 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.659579 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1414}
I20260928 12:33:22.680939 21686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } attempt: 3
Sep 28 12:33:22 dist-test-slave-g31s krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
I20260928 12:33:22.690456 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.681291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:33779 (local address 127.0.0.1:51008)
0928 12:33:22.681367 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:22.681382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:22.681453 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:22.681630 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:22.681631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:22.681916 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:22.681976 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.681980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.682940 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.682941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.683757 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:22.683761 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.684317 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.684319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:22.684384 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.684904 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:22.684909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:22.686516 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:22.689258 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.689260 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.689261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.689521 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.689722 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:22.689723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:22.689724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:22.689781 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:22.690130 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:22.690131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:22.690181 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 12:33:22.690235 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:33:22.690345 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 12:33:22.690812 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:22.681332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.8.13.60:33779)
0928 12:33:22.681376 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:22.681378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:22.681433 (+ 55us) server_negotiation.cc:407] Connection header received
0928 12:33:22.681534 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:22.681535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:22.681546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:22.681602 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:22.682060 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.682858 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.683869 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:22.684177 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:22.684249 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:22.686637 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:22.686640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:22.686640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:22.686650 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:22.689152 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.689600 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.689602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.689603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.689663 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:22.689899 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:22.689900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:22.689900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:22.690046 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:22.690103 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:22.690633 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 12:33:22.690718 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:33:22.692373 21686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0: Bootstrap starting.
I20260928 12:33:22.693058 21686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:22.693342 21686 log.cc:824] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:22.694170 21686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0: No bootstrap required, opened a new log
I20260928 12:33:22.696285 21686 raft_consensus.cc:348] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } }
I20260928 12:33:22.696437 21686 raft_consensus.cc:374] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:22.696470 21686 raft_consensus.cc:729] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79ecfc859f042c5951252d37045a9b0, State: Initialized, Role: FOLLOWER
I20260928 12:33:22.696619 21686 consensus_queue.cc:260] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } }
I20260928 12:33:22.696943 21803 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } } }
I20260928 12:33:22.697057 21803 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:33:22.696961 21686 sys_catalog.cc:564] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 12:33:22.700107 21814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:33:22.700172 21814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:33:22.702224 21618 master_runner.cc:263] No leader master found from master c79ecfc859f042c5951252d37045a9b0
I20260928 12:33:22.702275 21618 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 12:33:22.776386 21564 tablet.cc:2369] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:33:22.853456 21714 tablet.cc:2369] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:33:22.943138 21639 tablet.cc:2369] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:33:22.996127 21803 raft_consensus.cc:482] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:33:22.996260 21803 raft_consensus.cc:504] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } }
I20260928 12:33:22.996556 21803 leader_election.cc:288] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 333e637223c04f0a837a8cf5bdf579ca (127.8.13.62:35555), e0c54291811041338389e3fc125c1f7a (127.8.13.60:33779)
I20260928 12:33:22.996894 21585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c79ecfc859f042c5951252d37045a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333e637223c04f0a837a8cf5bdf579ca" is_pre_election: true
I20260928 12:33:22.996894 21735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c79ecfc859f042c5951252d37045a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0c54291811041338389e3fc125c1f7a" is_pre_election: true
I20260928 12:33:22.997076 21585 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c79ecfc859f042c5951252d37045a9b0 in term 0.
I20260928 12:33:22.997076 21735 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c79ecfc859f042c5951252d37045a9b0 in term 0.
I20260928 12:33:22.997292 21636 leader_election.cc:302] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333e637223c04f0a837a8cf5bdf579ca, c79ecfc859f042c5951252d37045a9b0; no voters:
I20260928 12:33:22.997478 21803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:33:22.997538 21803 raft_consensus.cc:482] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:33:22.997555 21803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:22.998342 21803 raft_consensus.cc:504] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } }
I20260928 12:33:22.998507 21803 leader_election.cc:288] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [CANDIDATE]: Term 1 election: Requested vote from peers 333e637223c04f0a837a8cf5bdf579ca (127.8.13.62:35555), e0c54291811041338389e3fc125c1f7a (127.8.13.60:33779)
I20260928 12:33:22.998730 21735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c79ecfc859f042c5951252d37045a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0c54291811041338389e3fc125c1f7a"
I20260928 12:33:22.998733 21585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c79ecfc859f042c5951252d37045a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333e637223c04f0a837a8cf5bdf579ca"
I20260928 12:33:22.998839 21735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:22.998842 21585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:22.999754 21585 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c79ecfc859f042c5951252d37045a9b0 in term 1.
I20260928 12:33:22.999754 21735 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c79ecfc859f042c5951252d37045a9b0 in term 1.
I20260928 12:33:22.999982 21637 leader_election.cc:302] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c79ecfc859f042c5951252d37045a9b0, e0c54291811041338389e3fc125c1f7a; no voters:
I20260928 12:33:23.000077 21803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:33:23.000188 21803 raft_consensus.cc:686] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [term 1 LEADER]: Becoming Leader. State: Replica: c79ecfc859f042c5951252d37045a9b0, State: Running, Role: LEADER
I20260928 12:33:23.000291 21803 consensus_queue.cc:237] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } }
I20260928 12:33:23.000710 21816 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c79ecfc859f042c5951252d37045a9b0. Latest consensus state: current_term: 1 leader_uuid: "c79ecfc859f042c5951252d37045a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } } }
I20260928 12:33:23.000792 21816 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:23.000994 21818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:33:23.001175 21818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:33:23.002557 21735 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [term 1 FOLLOWER]: Refusing update from remote peer c79ecfc859f042c5951252d37045a9b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:33:23.002599 21585 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [term 1 FOLLOWER]: Refusing update from remote peer c79ecfc859f042c5951252d37045a9b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:33:23.002858 21803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:33:23.002996 21816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:33:23.004040 21782 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c79ecfc859f042c5951252d37045a9b0. Latest consensus state: current_term: 1 leader_uuid: "c79ecfc859f042c5951252d37045a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } } }
I20260928 12:33:23.004062 21768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader c79ecfc859f042c5951252d37045a9b0. Latest consensus state: current_term: 1 leader_uuid: "c79ecfc859f042c5951252d37045a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } } }
I20260928 12:33:23.004137 21768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:33:23.004137 21782 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:33:23.004928 21803 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c79ecfc859f042c5951252d37045a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } } }
I20260928 12:33:23.004947 21816 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c79ecfc859f042c5951252d37045a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } } }
I20260928 12:33:23.005025 21803 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:23.005043 21816 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0 [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:23.005074 21819 mvcc.cc:205] Tried to move back new op lower bound from 7334292697096327168 to 7334292697089892352. Current Snapshot: MvccSnapshot[applied={T|T < 7334292697096327168}]
I20260928 12:33:23.005138 21768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c79ecfc859f042c5951252d37045a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } } }
I20260928 12:33:23.005200 21768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:33:23.005165 21782 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c79ecfc859f042c5951252d37045a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333e637223c04f0a837a8cf5bdf579ca" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35555 } } peers { permanent_uuid: "c79ecfc859f042c5951252d37045a9b0" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 42537 } } peers { permanent_uuid: "e0c54291811041338389e3fc125c1f7a" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 33779 } } }
I20260928 12:33:23.005229 21782 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:33:23.005273 21820 mvcc.cc:205] Tried to move back new op lower bound from 7334292697096327168 to 7334292697089892352. Current Snapshot: MvccSnapshot[applied={T|T < 7334292697096327168}]
I20260928 12:33:23.005285 21821 mvcc.cc:205] Tried to move back new op lower bound from 7334292697096327168 to 7334292697089892352. Current Snapshot: MvccSnapshot[applied={T|T < 7334292697096327168}]
I20260928 12:33:23.005650 21818 catalog_manager.cc:1347] Generated new cluster ID: 21d62a9112f14d5893945619f1e122c5
I20260928 12:33:23.005697 21818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:33:23.014650 21818 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:33:23.015259 21818 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:33:23.025398 21818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c79ecfc859f042c5951252d37045a9b0: Generated new TSK 0
I20260928 12:33:23.025650 21818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:33:23.642925 21779 catalog_manager.cc:1259] Loaded cluster ID: 21d62a9112f14d5893945619f1e122c5
I20260928 12:33:23.643026 21779 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca: loading cluster ID for follower catalog manager: success
I20260928 12:33:23.643962 21779 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca: acquiring CA information for follower catalog manager: success
I20260928 12:33:23.644279 21779 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 333e637223c04f0a837a8cf5bdf579ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:33:23.651000 21799 catalog_manager.cc:1259] Loaded cluster ID: 21d62a9112f14d5893945619f1e122c5
I20260928 12:33:23.651087 21799 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a: loading cluster ID for follower catalog manager: success
I20260928 12:33:23.651913 21799 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a: acquiring CA information for follower catalog manager: success
I20260928 12:33:23.652251 21799 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0c54291811041338389e3fc125c1f7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:33:24.072010 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 21543
I20260928 12:33:24.076771 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 21618
I20260928 12:33:24.081470 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 21693
2026-09-28T12:33:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2997 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:33:24 dist-test-slave-g31s krb5kdc[21834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:33:24 dist-test-slave-g31s krb5kdc[21834](info): set up 2 sockets
Sep 28 12:33:24 dist-test-slave-g31s krb5kdc[21834](info): commencing operation
krb5kdc: starting...
W20260928 12:33:25.029562 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:33:25 dist-test-slave-g31s krb5kdc[21834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:33:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:33:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:33:25.149329 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:39595
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:39595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:25.229712 21847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:25.229956 21847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:25.229979 21847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:25.231165 21847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:33:25.231213 21847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:25.231227 21847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:25.231249 21847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:33:25.231272 21847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:33:25.232796 21847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:39595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:39595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:25.233065 21847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:25.233291 21847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:25.238322 21853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:25.238350 21852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:25.238449 21855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:25.238529 21847 server_base.cc:1034] running on GCE node
I20260928 12:33:25.238761 21847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:25.239074 21847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:25.240224 21847 hybrid_clock.cc:648] HybridClock initialized: now 1790598805240207 us; error 31 us; skew 500 ppm
Sep 28 12:33:25 dist-test-slave-g31s krb5kdc[21834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598805, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:25.242715 21847 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:33:25.243176 21847 webserver.cc:460] Webserver started at http://127.8.13.62:42333/ using document root <none> and password file <none>
I20260928 12:33:25.243388 21847 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:25.243435 21847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:25.243593 21847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:25.244705 21847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "9df928cf403b410ea098914379956822"
format_stamp: "Formatted at 2026-09-28 12:33:25 on dist-test-slave-g31s"
server_key: "c474f81545cd9fd59e07c684562d6aba"
server_key_iv: "85e559437467b61d54e18f464cd140ca"
server_key_version: "encryptionkey@0"
I20260928 12:33:25.245057 21847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "9df928cf403b410ea098914379956822"
format_stamp: "Formatted at 2026-09-28 12:33:25 on dist-test-slave-g31s"
server_key: "c474f81545cd9fd59e07c684562d6aba"
server_key_iv: "85e559437467b61d54e18f464cd140ca"
server_key_version: "encryptionkey@0"
I20260928 12:33:25.246477 21847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:33:25.247288 21862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:25.247481 21847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:25.247581 21847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "9df928cf403b410ea098914379956822"
format_stamp: "Formatted at 2026-09-28 12:33:25 on dist-test-slave-g31s"
server_key: "c474f81545cd9fd59e07c684562d6aba"
server_key_iv: "85e559437467b61d54e18f464cd140ca"
server_key_version: "encryptionkey@0"
I20260928 12:33:25.247655 21847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:25.258113 21847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:25.260085 21847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:25.260222 21847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:25.264328 21847 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:39595
I20260928 12:33:25.264360 21914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:39595 every 8 connection(s)
I20260928 12:33:25.264767 21847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:33:25.265661 21915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:25.268044 21915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822: Bootstrap starting.
I20260928 12:33:25.268628 21915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:25.268864 21915 log.cc:824] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:25.269572 21915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822: No bootstrap required, opened a new log
I20260928 12:33:25.271212 21915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df928cf403b410ea098914379956822" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39595 } }
I20260928 12:33:25.271369 21915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:25.271392 21915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df928cf403b410ea098914379956822, State: Initialized, Role: FOLLOWER
I20260928 12:33:25.271510 21915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df928cf403b410ea098914379956822" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39595 } }
I20260928 12:33:25.271567 21915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:33:25.271595 21915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:33:25.271641 21915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:25.272207 21915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df928cf403b410ea098914379956822" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39595 } }
I20260928 12:33:25.272305 21915 leader_election.cc:302] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9df928cf403b410ea098914379956822; no voters:
I20260928 12:33:25.272482 21915 leader_election.cc:288] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:33:25.272557 21918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:33:25.272727 21918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [term 1 LEADER]: Becoming Leader. State: Replica: 9df928cf403b410ea098914379956822, State: Running, Role: LEADER
I20260928 12:33:25.272743 21915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:33:25.272850 21918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df928cf403b410ea098914379956822" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39595 } }
I20260928 12:33:25.273445 21920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df928cf403b410ea098914379956822. Latest consensus state: current_term: 1 leader_uuid: "9df928cf403b410ea098914379956822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df928cf403b410ea098914379956822" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39595 } } }
I20260928 12:33:25.273543 21920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:25.273681 21919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9df928cf403b410ea098914379956822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df928cf403b410ea098914379956822" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39595 } } }
I20260928 12:33:25.273746 21919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822 [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:25.273973 21927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:33:25.274250 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 21847
I20260928 12:33:25.274358 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:33:25.274571 8244 external_mini_cluster.cc:1428] Setting key ee5ed23f6fe7b5ffb42decae7c074090
I20260928 12:33:25.274573 21927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:33:25.276167 21927 catalog_manager.cc:1347] Generated new cluster ID: e42200c2e5a148d8bb154aa2e5b447e3
I20260928 12:33:25.276221 21927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:33:25 dist-test-slave-g31s krb5kdc[21834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:25.286695 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:25.275032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.8.13.62:39595)
0928 12:33:25.275207 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:25.275218 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:25.275318 (+ 100us) server_negotiation.cc:407] Connection header received
0928 12:33:25.275491 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:25.275498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:25.275577 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:25.275718 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:25.276476 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:25.278026 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:25.279092 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:25.279765 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:25.279891 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:25.282869 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:25.282876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:25.282876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:25.282893 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:25.284920 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:25.285371 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:25.285374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:25.285374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:25.285498 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:25.285681 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:25.285681 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:25.285681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:25.286031 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:25.286201 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:25.286283 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 12:33:25.286443 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260928 12:33:25.287667 21927 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:33:25.287715 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:33:25.288376 21927 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:33:25.290508 21927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9df928cf403b410ea098914379956822: Generated new TSK 0
I20260928 12:33:25.290674 21927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:33:25.341929 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 21847
2026-09-28T12:33:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1258 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:33:25 dist-test-slave-g31s krb5kdc[21943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:33:25 dist-test-slave-g31s krb5kdc[21943](info): set up 2 sockets
Sep 28 12:33:25 dist-test-slave-g31s krb5kdc[21943](info): commencing operation
krb5kdc: starting...
W20260928 12:33:26.285609 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:33:26 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:33:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:33:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:33:26.401319 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:34429
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:34429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:26.482455 21956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:26.482682 21956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:26.482707 21956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:26.483915 21956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:33:26.483963 21956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:26.483978 21956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:26.483989 21956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:33:26.483999 21956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:33:26.485569 21956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:34429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:34429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:26.485792 21956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:26.486030 21956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:33:26.490953 21956 server_base.cc:1034] running on GCE node
W20260928 12:33:26.490890 21964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:26.490911 21961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:26.490911 21962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:26.491374 21956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:26.491690 21956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:26.492864 21956 hybrid_clock.cc:648] HybridClock initialized: now 1790598806492844 us; error 37 us; skew 500 ppm
Sep 28 12:33:26 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598806, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:26.495385 21956 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:33:26.495857 21956 webserver.cc:460] Webserver started at http://127.8.13.62:43919/ using document root <none> and password file <none>
I20260928 12:33:26.496071 21956 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:26.496117 21956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:26.496236 21956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:26.497303 21956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "305f40b26cf5e29dd46780e0c9e59c34"
server_key_iv: "fe15013719df15a861971cb4a9213131"
server_key_version: "encryptionkey@0"
I20260928 12:33:26.497648 21956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "305f40b26cf5e29dd46780e0c9e59c34"
server_key_iv: "fe15013719df15a861971cb4a9213131"
server_key_version: "encryptionkey@0"
I20260928 12:33:26.499003 21956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:26.499729 21971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:26.499890 21956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:33:26.499980 21956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "305f40b26cf5e29dd46780e0c9e59c34"
server_key_iv: "fe15013719df15a861971cb4a9213131"
server_key_version: "encryptionkey@0"
I20260928 12:33:26.500046 21956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:26.518895 21956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:26.521000 21956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:26.521140 21956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:26.524765 21956 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:34429
I20260928 12:33:26.524830 22023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:34429 every 8 connection(s)
I20260928 12:33:26.525130 21956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:33:26.526057 22024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:26.526050 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 21956
I20260928 12:33:26.526141 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:33:26.526310 8244 external_mini_cluster.cc:1428] Setting key 1a756a9846dfc8b7fe4daacae3cfb61e
I20260928 12:33:26.529150 22024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Bootstrap starting.
I20260928 12:33:26.529851 22024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:26.530112 22024 log.cc:824] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:26.530751 22024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: No bootstrap required, opened a new log
Sep 28 12:33:26 dist-test-slave-g31s krb5kdc[21943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:26.532756 22024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:26.532923 22024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:26.532970 22024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf67fd34ea14838a7ff9da02ffa1d3d, State: Initialized, Role: FOLLOWER
I20260928 12:33:26.533097 22024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:26.533166 22024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:33:26.533187 22024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:33:26.533231 22024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:26.533936 22024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:26.534042 22024 leader_election.cc:302] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cf67fd34ea14838a7ff9da02ffa1d3d; no voters:
I20260928 12:33:26.534207 22024 leader_election.cc:288] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:33:26.534332 22029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:33:26.534457 22024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:33:26.534526 22029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 1 LEADER]: Becoming Leader. State: Replica: 9cf67fd34ea14838a7ff9da02ffa1d3d, State: Running, Role: LEADER
I20260928 12:33:26.534662 22029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:26.535012 22030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } } }
I20260928 12:33:26.535121 22030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:26.535012 22031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cf67fd34ea14838a7ff9da02ffa1d3d. Latest consensus state: current_term: 1 leader_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } } }
I20260928 12:33:26.535374 22031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:26.535738 22038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:33:26.536239 22038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:33:26.536918 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:26.526791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.8.13.62:34429)
0928 12:33:26.526970 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:26.526979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:26.527012 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:33:26.527486 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:26.527496 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:26.527559 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:26.527705 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:26.528230 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:26.529590 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:26.530354 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:26.530835 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:26.530936 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:26.533315 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:26.533326 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:26.533327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:26.533337 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:26.535338 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:26.535771 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:26.535774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:26.535775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:26.535870 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:26.536087 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:26.536089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:26.536089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:26.536331 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:26.536519 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:26.536586 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 12:33:26.536718 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260928 12:33:26.538004 22038 catalog_manager.cc:1347] Generated new cluster ID: b16abdb8938b440ca4bb41c2db49113f
I20260928 12:33:26.538046 22038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:33:26.561669 22038 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:33:26.562256 22038 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:33:26.573832 22038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Generated new TSK 0
I20260928 12:33:26.574056 22038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:33:26.609133 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34429
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:26.716339 22050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:26.716542 22050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:26.716565 22050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:26.717681 22050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:26.717720 22050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:26.717744 22050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:33:26.719170 22050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:26.719352 22050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:26.719529 22050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:26.724160 22055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:26.724164 22056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:26.724165 22058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:26.724263 22050 server_base.cc:1034] running on GCE node
I20260928 12:33:26.724550 22050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:26.724748 22050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:26.725844 22050 hybrid_clock.cc:648] HybridClock initialized: now 1790598806725832 us; error 29 us; skew 500 ppm
Sep 28 12:33:26 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598806, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:26.728173 22050 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:33:26.728585 22050 webserver.cc:460] Webserver started at http://127.8.13.1:33547/ using document root <none> and password file <none>
I20260928 12:33:26.728775 22050 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:26.728813 22050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:26.728904 22050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:26.729951 22050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "15322d840f904ab8b3d71afe43387c80"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "a237748e221ce9a6bdadb7ecd62de345"
server_key_iv: "8a27d713dc435705e2f5b4b416688942"
server_key_version: "encryptionkey@0"
I20260928 12:33:26.730244 22050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "15322d840f904ab8b3d71afe43387c80"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "a237748e221ce9a6bdadb7ecd62de345"
server_key_iv: "8a27d713dc435705e2f5b4b416688942"
server_key_version: "encryptionkey@0"
I20260928 12:33:26.731350 22050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:26.731953 22065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:26.732118 22050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:33:26.732182 22050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "15322d840f904ab8b3d71afe43387c80"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "a237748e221ce9a6bdadb7ecd62de345"
server_key_iv: "8a27d713dc435705e2f5b4b416688942"
server_key_version: "encryptionkey@0"
I20260928 12:33:26.732257 22050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:26.742657 22050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:26.744975 22050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:26.745115 22050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:26.745347 22050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:26.745666 22050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:26.745702 22050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:26.745728 22050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:26.745752 22050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:26.751372 22050 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:39173
I20260928 12:33:26.751437 22178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:39173 every 8 connection(s)
I20260928 12:33:26.751754 22050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:33:26.754621 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 22050
I20260928 12:33:26.754716 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:33:26.754909 8244 external_mini_cluster.cc:1428] Setting key 881d5ea40836c38c97879dc6fc07c96f
Sep 28 12:33:26 dist-test-slave-g31s krb5kdc[21943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598806, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:26.761703 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:26.752516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:50117 (local address 127.8.13.62:34429)
0928 12:33:26.752566 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:26.752568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:26.752774 (+ 206us) server_negotiation.cc:407] Connection header received
0928 12:33:26.753244 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:26.753246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:26.753256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:26.753305 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:26.754316 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:26.754873 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:26.756279 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:26.756489 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:26.756544 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:26.758904 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:26.758906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:26.758906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:26.758913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:26.760209 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:26.760715 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:26.760716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:26.760716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:26.760765 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:26.761047 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:26.761047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:26.761048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:26.761149 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:26.761193 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:26.761562 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 12:33:26.761631 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:33:26.762029 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:26.752498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:50117)
0928 12:33:26.752657 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:26.752694 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:26.753167 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:26.753362 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:26.753365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:26.754014 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:26.754222 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:26.754227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:26.754969 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:26.754970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:26.756198 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:26.756202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:26.756565 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:26.756566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:26.756705 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:26.757306 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:26.757315 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:26.758808 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:26.760294 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:26.760305 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:26.760307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:26.760641 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:26.760823 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:26.760823 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:26.760824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:26.760984 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:26.761201 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:26.761207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:26.761410 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 12:33:26.761539 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 12:33:26.761724 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260928 12:33:26.762552 22179 heartbeater.cc:347] Connected to a master server at 127.8.13.62:34429
I20260928 12:33:26.762660 22179 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:26.762887 22179 heartbeater.cc:511] Master 127.8.13.62:34429 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
I20260928 12:33:26.763432 21988 ts_manager.cc:194] Registered new tserver with Master: 15322d840f904ab8b3d71afe43387c80 (127.8.13.1:39173)
I20260928 12:33:26.764058 21988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:50117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:33:26.781919 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34429
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:26.862308 22184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:26.862541 22184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:26.862576 22184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:26.863850 22184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:26.863904 22184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:26.863945 22184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:33:26.865543 22184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:26.865849 22184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:26.866098 22184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:26.871131 22190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:26.871295 22184 server_base.cc:1034] running on GCE node
W20260928 12:33:26.871140 22192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:26.871171 22189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:26.871639 22184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:26.871888 22184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:26.873035 22184 hybrid_clock.cc:648] HybridClock initialized: now 1790598806873036 us; error 40 us; skew 500 ppm
Sep 28 12:33:26 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598806, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:26.875546 22184 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:33:26.875998 22184 webserver.cc:460] Webserver started at http://127.8.13.2:39417/ using document root <none> and password file <none>
I20260928 12:33:26.876204 22184 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:26.876266 22184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:26.876379 22184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:26.877417 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "629d0005d3e74139a1f06e3f829db5e2"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "5d9ac410fdcd5c02088fe94b444da3ca"
server_key_iv: "7b78b69a82e211c3d92a103d3f8ad619"
server_key_version: "encryptionkey@0"
I20260928 12:33:26.877718 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "629d0005d3e74139a1f06e3f829db5e2"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "5d9ac410fdcd5c02088fe94b444da3ca"
server_key_iv: "7b78b69a82e211c3d92a103d3f8ad619"
server_key_version: "encryptionkey@0"
I20260928 12:33:26.879024 22184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:33:26.879711 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:26.879861 22184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:33:26.879945 22184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "629d0005d3e74139a1f06e3f829db5e2"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "5d9ac410fdcd5c02088fe94b444da3ca"
server_key_iv: "7b78b69a82e211c3d92a103d3f8ad619"
server_key_version: "encryptionkey@0"
I20260928 12:33:26.880028 22184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:26.899134 22184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:26.901068 22184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:26.901216 22184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:26.901443 22184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:26.901750 22184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:26.901786 22184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:26.901853 22184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:26.901877 22184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:26.907660 22184 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:37421
I20260928 12:33:26.907714 22312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:37421 every 8 connection(s)
I20260928 12:33:26.908028 22184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:33:26 dist-test-slave-g31s krb5kdc[21943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598806, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:26.916805 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:26.908714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:41137 (local address 127.8.13.62:34429)
0928 12:33:26.908775 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:26.908777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:26.908951 (+ 174us) server_negotiation.cc:407] Connection header received
0928 12:33:26.909464 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:26.909465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:26.909478 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:26.909522 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:26.910470 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:26.910968 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:26.911936 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:26.912134 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:26.912191 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:26.914250 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:26.914252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:26.914252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:26.914259 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:26.915459 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:26.915857 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:26.915857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:26.915858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:26.915902 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:26.916140 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:26.916141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:26.916142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:26.916265 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:26.916328 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:26.916656 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 12:33:26.916735 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:33:26.916935 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 22184
I20260928 12:33:26.916996 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:26.908693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:41137)
0928 12:33:26.908837 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:26.908872 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:26.909366 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:26.909589 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:26.909593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:26.910177 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:26.910369 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:26.910373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:26.911053 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:26.911054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:26.911852 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:26.911855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:26.912204 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:26.912205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:26.912319 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:26.912801 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:26.912809 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:26.914161 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:26.915549 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:26.915556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:26.915558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:26.915792 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:26.915949 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:26.915949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:26.915949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:26.916044 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:26.916352 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:26.916359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:26.916508 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 12:33:26.916598 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 12:33:26.916756 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260928 12:33:26.917030 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:33:26.917230 8244 external_mini_cluster.cc:1428] Setting key 77b0ee3ad7e7762822a5c3616e6789e0
I20260928 12:33:26.917485 22313 heartbeater.cc:347] Connected to a master server at 127.8.13.62:34429
I20260928 12:33:26.917589 22313 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:26.917761 22313 heartbeater.cc:511] Master 127.8.13.62:34429 requested a full tablet report, sending...
I20260928 12:33:26.918128 21988 ts_manager.cc:194] Registered new tserver with Master: 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421)
I20260928 12:33:26.918596 21988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:41137
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:33:26.940599 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34429
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:27.020593 22318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:27.020794 22318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:27.020824 22318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:27.021960 22318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:27.022011 22318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:27.022044 22318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:33:27.023434 22318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:27.023619 22318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:27.023793 22318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:27.028451 22323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:27.028469 22324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:27.028548 22326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:27.028581 22318 server_base.cc:1034] running on GCE node
I20260928 12:33:27.028812 22318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:27.029016 22318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:27.030148 22318 hybrid_clock.cc:648] HybridClock initialized: now 1790598807030136 us; error 30 us; skew 500 ppm
Sep 28 12:33:27 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598807, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:27.032444 22318 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:33:27.032883 22318 webserver.cc:460] Webserver started at http://127.8.13.3:33099/ using document root <none> and password file <none>
I20260928 12:33:27.033079 22318 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:27.033128 22318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:27.033252 22318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:27.034353 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "f158271212e04ec58df6d46ef2bbbd62"
format_stamp: "Formatted at 2026-09-28 12:33:27 on dist-test-slave-g31s"
server_key: "e021115452ea73dd1d5a1c48c2f1203c"
server_key_iv: "9b7b9f84992cfc59ea6cba9d093cadf2"
server_key_version: "encryptionkey@0"
I20260928 12:33:27.034724 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "f158271212e04ec58df6d46ef2bbbd62"
format_stamp: "Formatted at 2026-09-28 12:33:27 on dist-test-slave-g31s"
server_key: "e021115452ea73dd1d5a1c48c2f1203c"
server_key_iv: "9b7b9f84992cfc59ea6cba9d093cadf2"
server_key_version: "encryptionkey@0"
I20260928 12:33:27.036190 22318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:33:27.036981 22333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.037127 22318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:33:27.037307 22318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "f158271212e04ec58df6d46ef2bbbd62"
format_stamp: "Formatted at 2026-09-28 12:33:27 on dist-test-slave-g31s"
server_key: "e021115452ea73dd1d5a1c48c2f1203c"
server_key_iv: "9b7b9f84992cfc59ea6cba9d093cadf2"
server_key_version: "encryptionkey@0"
I20260928 12:33:27.037420 22318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:27.057361 22318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:27.059221 22318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:27.059335 22318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:27.059576 22318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:27.059890 22318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:27.059923 22318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.059947 22318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:27.059970 22318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.065428 22318 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:37479
I20260928 12:33:27.065465 22446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:37479 every 8 connection(s)
I20260928 12:33:27.065783 22318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:33:27 dist-test-slave-g31s krb5kdc[21943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598807, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:27.074833 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.066532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:38253 (local address 127.8.13.62:34429)
0928 12:33:27.066601 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.066604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.066793 (+ 189us) server_negotiation.cc:407] Connection header received
0928 12:33:27.067309 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.067311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.067323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.067374 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:27.068387 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.068879 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.069924 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.070114 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.070169 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.072201 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:27.072203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:27.072203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:27.072212 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:27.073288 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.073703 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.073705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.073705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.073773 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.074111 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.074111 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.074112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.074235 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:27.074312 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.074643 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.074737 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:33:27.075014 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.066462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:38253)
0928 12:33:27.066667 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.066707 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.067202 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.067481 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.067484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.068068 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:27.068279 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.068284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.068978 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.068979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.069793 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.069839 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.070186 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.070187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.070303 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.070774 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:27.070782 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:27.072105 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:27.073366 (+ 1261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:27.073373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:27.073376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:27.073632 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:27.073911 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:27.073912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:27.073912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:27.074047 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:27.074321 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:27.074329 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:27.074513 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.074609 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.074781 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":90,"threads_started":1}
I20260928 12:33:27.075428 22447 heartbeater.cc:347] Connected to a master server at 127.8.13.62:34429
I20260928 12:33:27.075533 22447 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:27.075680 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 22318
I20260928 12:33:27.075711 22447 heartbeater.cc:511] Master 127.8.13.62:34429 requested a full tablet report, sending...
I20260928 12:33:27.075750 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:33:27.075958 8244 external_mini_cluster.cc:1428] Setting key ca0b3b7e78c059f737703662e8db0a16
I20260928 12:33:27.076116 21988 ts_manager.cc:194] Registered new tserver with Master: f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479)
I20260928 12:33:27.076601 21988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:38253
I20260928 12:33:27.076828 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:33:27.083847 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.078226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.8.13.62:34429)
0928 12:33:27.078264 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.078266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.078383 (+ 117us) server_negotiation.cc:407] Connection header received
0928 12:33:27.078443 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.078443 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.078453 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.078497 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:27.078995 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.079467 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.080150 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.080336 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.080388 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.081511 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:27.081513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:27.081513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:27.081521 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:27.082804 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.083120 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.083121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.083121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.083171 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.083333 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.083334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.083334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.083482 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:27.083570 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.083676 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.083753 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:33:27.085461 21988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:33:27.087153 21988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:33:27.093853 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.089641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.8.13.2:37421)
0928 12:33:27.089865 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.089871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.089985 (+ 114us) server_negotiation.cc:407] Connection header received
0928 12:33:27.090079 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.090082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.090099 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.090175 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:27.090357 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.091193 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.092550 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.093365 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.093446 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.093516 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.093686 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.093735 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260928 12:33:27.093880 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.089710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39173 (local address 127.0.0.1:57576)
0928 12:33:27.090373 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.090382 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.090427 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.090625 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.090626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.090629 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:27.090666 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.090667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.091838 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.091838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.092644 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.092647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.093549 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.093550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.093599 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.093603 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.093645 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.093724 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":16,"thread_start_us":56,"threads_started":1}
I20260928 12:33:27.093994 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.089659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.8.13.3:37479)
0928 12:33:27.090127 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.090131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.090139 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:33:27.090178 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.090181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.090194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.090285 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:27.090437 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.091540 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.092822 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.093523 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.093579 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.093628 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.093857 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.093914 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":72,"threads_started":1}
I20260928 12:33:27.094028 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.089513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:37479 (local address 127.0.0.1:32920)
0928 12:33:27.089858 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.089920 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.089976 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.090289 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.090290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.090292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:27.090348 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.090350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.091648 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.091649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.092749 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.092751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.093593 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.093595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.093667 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.093671 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.093718 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.093762 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":34,"thread_start_us":58,"threads_started":1}
I20260928 12:33:27.094121 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.089517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:37421 (local address 127.0.0.1:49288)
0928 12:33:27.089859 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.089903 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.089975 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.090187 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.090190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.090192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:27.090271 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.090273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.091274 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.091274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.092457 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.092460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.093455 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.093457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.093547 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.093561 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.093660 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.093724 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":21,"thread_start_us":95,"threads_started":1}
I20260928 12:33:27.094215 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.089853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.8.13.1:39173)
0928 12:33:27.090076 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.090080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.090480 (+ 400us) server_negotiation.cc:407] Connection header received
0928 12:33:27.090523 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.090525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.090539 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.090612 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:27.090744 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.091751 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.092738 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.093385 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.093458 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.093518 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.093776 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.093852 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":94,"threads_started":1}
I20260928 12:33:27.094965 22247 tablet_service.cc:1467] Processing CreateTablet for tablet 00c5f4ece9b34bba910a828880252d0f (DEFAULT_TABLE table=test-table [id=e073cedae0d9404ca98f323aabada83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:27.095104 22113 tablet_service.cc:1467] Processing CreateTablet for tablet 00c5f4ece9b34bba910a828880252d0f (DEFAULT_TABLE table=test-table [id=e073cedae0d9404ca98f323aabada83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:27.095104 22381 tablet_service.cc:1467] Processing CreateTablet for tablet 00c5f4ece9b34bba910a828880252d0f (DEFAULT_TABLE table=test-table [id=e073cedae0d9404ca98f323aabada83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:27.095364 22247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c5f4ece9b34bba910a828880252d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:27.095438 22381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c5f4ece9b34bba910a828880252d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:27.095444 22113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c5f4ece9b34bba910a828880252d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:27.098349 22464 tablet_bootstrap.cc:492] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: Bootstrap starting.
I20260928 12:33:27.098670 22466 tablet_bootstrap.cc:492] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Bootstrap starting.
I20260928 12:33:27.099002 22464 tablet_bootstrap.cc:654] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:27.099169 22466 tablet_bootstrap.cc:654] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:27.099470 22464 log.cc:824] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:27.099552 22466 log.cc:824] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:27.099840 22465 tablet_bootstrap.cc:492] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: Bootstrap starting.
I20260928 12:33:27.100250 22466 tablet_bootstrap.cc:492] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: No bootstrap required, opened a new log
I20260928 12:33:27.100250 22464 tablet_bootstrap.cc:492] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: No bootstrap required, opened a new log
I20260928 12:33:27.100317 22464 ts_tablet_manager.cc:1397] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:33:27.100359 22466 ts_tablet_manager.cc:1397] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:33:27.100360 22465 tablet_bootstrap.cc:654] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:27.100780 22465 log.cc:824] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:27.101388 22465 tablet_bootstrap.cc:492] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: No bootstrap required, opened a new log
I20260928 12:33:27.101449 22465 ts_tablet_manager.cc:1397] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:33:27.102304 22464 raft_consensus.cc:348] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.102492 22464 raft_consensus.cc:374] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:27.102538 22464 raft_consensus.cc:729] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629d0005d3e74139a1f06e3f829db5e2, State: Initialized, Role: FOLLOWER
I20260928 12:33:27.102557 22466 raft_consensus.cc:348] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.102655 22464 consensus_queue.cc:260] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.102717 22466 raft_consensus.cc:374] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:27.102757 22466 raft_consensus.cc:729] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15322d840f904ab8b3d71afe43387c80, State: Initialized, Role: FOLLOWER
I20260928 12:33:27.102869 22466 consensus_queue.cc:260] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.102981 22464 ts_tablet_manager.cc:1428] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:33:27.103108 22466 ts_tablet_manager.cc:1428] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:33:27.103121 22313 heartbeater.cc:503] Master 127.8.13.62:34429 was elected leader, sending a full tablet report...
I20260928 12:33:27.103216 22179 heartbeater.cc:503] Master 127.8.13.62:34429 was elected leader, sending a full tablet report...
I20260928 12:33:27.103114 22465 raft_consensus.cc:348] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.103266 22465 raft_consensus.cc:374] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:27.103294 22465 raft_consensus.cc:729] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f158271212e04ec58df6d46ef2bbbd62, State: Initialized, Role: FOLLOWER
I20260928 12:33:27.103500 22465 consensus_queue.cc:260] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.103767 22465 ts_tablet_manager.cc:1428] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:33:27.103820 22447 heartbeater.cc:503] Master 127.8.13.62:34429 was elected leader, sending a full tablet report...
W20260928 12:33:27.158595 22314 tablet.cc:2369] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:33:27.252519 22180 tablet.cc:2369] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:33:27.308133 22470 raft_consensus.cc:482] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:33:27.308261 22470 raft_consensus.cc:504] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.308569 22470 leader_election.cc:288] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479), 15322d840f904ab8b3d71afe43387c80 (127.8.13.1:39173)
I20260928 12:33:27.311365 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.308972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:37969 (local address 127.8.13.1:39173)
0928 12:33:27.309015 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.309017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.309095 (+ 78us) server_negotiation.cc:407] Connection header received
0928 12:33:27.309139 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.309139 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.309150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.309201 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:27.309350 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.309831 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.310726 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.311070 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.311124 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.311135 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.311262 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.311299 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:33:27.311398 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.308820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:39467 (local address 127.8.13.3:37479)
0928 12:33:27.308875 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.308877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.308887 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:33:27.308966 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.308967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.308979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.309023 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:27.309212 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.309736 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.310708 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.311075 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.311133 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.311143 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.311292 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.311331 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:33:27.311405 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.308750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:37479 (local address 127.8.13.2:39467)
0928 12:33:27.308803 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.308814 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.308876 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.309071 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.309071 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.309073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:27.309142 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.309144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.309827 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.309828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.310629 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.310631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.311150 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.311151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.311228 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.311231 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.311282 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.311316 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":45}
I20260928 12:33:27.311414 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.308869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39173 (local address 127.8.13.2:37969)
0928 12:33:27.309033 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.309041 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.309096 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.309211 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.309211 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.309213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:27.309278 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.309279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.309896 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.309896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.310646 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.310648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.311136 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.311137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.311206 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.311209 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.311248 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.311285 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260928 12:33:27.311841 22401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c5f4ece9b34bba910a828880252d0f" candidate_uuid: "629d0005d3e74139a1f06e3f829db5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f158271212e04ec58df6d46ef2bbbd62" is_pre_election: true
I20260928 12:33:27.311853 22133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c5f4ece9b34bba910a828880252d0f" candidate_uuid: "629d0005d3e74139a1f06e3f829db5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15322d840f904ab8b3d71afe43387c80" is_pre_election: true
I20260928 12:33:27.311975 22401 raft_consensus.cc:2454] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 629d0005d3e74139a1f06e3f829db5e2 in term 0.
I20260928 12:33:27.311975 22133 raft_consensus.cc:2454] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 629d0005d3e74139a1f06e3f829db5e2 in term 0.
I20260928 12:33:27.312191 22200 leader_election.cc:302] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15322d840f904ab8b3d71afe43387c80, 629d0005d3e74139a1f06e3f829db5e2; no voters:
I20260928 12:33:27.312307 22470 raft_consensus.cc:2781] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:33:27.312363 22470 raft_consensus.cc:482] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:33:27.312379 22470 raft_consensus.cc:3037] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:27.313061 22470 raft_consensus.cc:504] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.313174 22470 leader_election.cc:288] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [CANDIDATE]: Term 1 election: Requested vote from peers f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479), 15322d840f904ab8b3d71afe43387c80 (127.8.13.1:39173)
I20260928 12:33:27.313341 22401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c5f4ece9b34bba910a828880252d0f" candidate_uuid: "629d0005d3e74139a1f06e3f829db5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f158271212e04ec58df6d46ef2bbbd62"
I20260928 12:33:27.313364 22133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c5f4ece9b34bba910a828880252d0f" candidate_uuid: "629d0005d3e74139a1f06e3f829db5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15322d840f904ab8b3d71afe43387c80"
I20260928 12:33:27.313428 22401 raft_consensus.cc:3037] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:27.313427 22133 raft_consensus.cc:3037] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:27.314286 22133 raft_consensus.cc:2454] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 629d0005d3e74139a1f06e3f829db5e2 in term 1.
I20260928 12:33:27.314286 22401 raft_consensus.cc:2454] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 629d0005d3e74139a1f06e3f829db5e2 in term 1.
I20260928 12:33:27.314450 22200 leader_election.cc:302] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 629d0005d3e74139a1f06e3f829db5e2, f158271212e04ec58df6d46ef2bbbd62; no voters:
I20260928 12:33:27.314538 22470 raft_consensus.cc:2781] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:33:27.314680 22470 raft_consensus.cc:686] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 1 LEADER]: Becoming Leader. State: Replica: 629d0005d3e74139a1f06e3f829db5e2, State: Running, Role: LEADER
I20260928 12:33:27.314762 22470 consensus_queue.cc:237] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.315608 21987 catalog_manager.cc:5495] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2). New cstate: current_term: 1 leader_uuid: "629d0005d3e74139a1f06e3f829db5e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } health_report { overall_health: UNKNOWN } } }
W20260928 12:33:27.316254 22448 tablet.cc:2369] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:33:27.376016 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.373785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.8.13.2:37421)
0928 12:33:27.373843 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.373845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.373855 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:33:27.373926 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.373927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.373951 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.374019 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:33:27.374136 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.374630 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.375340 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.375555 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.375610 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.375720 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:33:27.375828 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:33:27.375892 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.375904 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.375923 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:33:27.377756 22401 raft_consensus.cc:1261] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62 [term 1 FOLLOWER]: Refusing update from remote peer 629d0005d3e74139a1f06e3f829db5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:33:27.377756 22133 raft_consensus.cc:1261] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Refusing update from remote peer 629d0005d3e74139a1f06e3f829db5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:33:27.378108 22470 consensus_queue.cc:1035] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:33:27.378261 22474 consensus_queue.cc:1035] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:33:27.380851 22478 mvcc.cc:205] Tried to move back new op lower bound from 7334292715016601600 to 7334292714761781248. Current Snapshot: MvccSnapshot[applied={T|T < 7334292715016601600}]
I20260928 12:33:27.382776 21987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:33:27.382874 21987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:33:27.383615 21987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Sending DeleteTablet for 3 replicas of tablet 00c5f4ece9b34bba910a828880252d0f
I20260928 12:33:27.383962 22247 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c5f4ece9b34bba910a828880252d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:27 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:49288
I20260928 12:33:27.383960 22113 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c5f4ece9b34bba910a828880252d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:27 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:57576
I20260928 12:33:27.384164 22381 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c5f4ece9b34bba910a828880252d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:27 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:32920
I20260928 12:33:27.384239 22487 tablet_replica.cc:330] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: stopping tablet replica
I20260928 12:33:27.384335 22487 raft_consensus.cc:2229] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:33:27.384395 22489 tablet_replica.cc:330] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: stopping tablet replica
I20260928 12:33:27.384490 22487 raft_consensus.cc:2258] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:33:27.384536 22489 raft_consensus.cc:2229] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:33:27.384546 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 22050
I20260928 12:33:27.384660 22489 raft_consensus.cc:2258] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:33:27.385507 22489 ts_tablet_manager.cc:1905] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:33:27.385957 22488 tablet_replica.cc:330] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: stopping tablet replica
I20260928 12:33:27.386098 22488 raft_consensus.cc:2229] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:33:27.386265 22488 raft_consensus.cc:2258] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:33:27.386771 22488 ts_tablet_manager.cc:1905] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:33:27.387454 22489 ts_tablet_manager.cc:1918] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:33:27.387591 22489 log.cc:1196] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/00c5f4ece9b34bba910a828880252d0f
I20260928 12:33:27.387924 22489 ts_tablet_manager.cc:1939] T 00c5f4ece9b34bba910a828880252d0f P f158271212e04ec58df6d46ef2bbbd62: Deleting consensus metadata
I20260928 12:33:27.388335 22488 ts_tablet_manager.cc:1918] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:33:27.388461 22488 log.cc:1196] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/00c5f4ece9b34bba910a828880252d0f
I20260928 12:33:27.388420 21972 catalog_manager.cc:4841] TS f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479): tablet 00c5f4ece9b34bba910a828880252d0f (table test-table [id=e073cedae0d9404ca98f323aabada83a]) successfully deleted
I20260928 12:33:27.388742 22488 ts_tablet_manager.cc:1939] T 00c5f4ece9b34bba910a828880252d0f P 629d0005d3e74139a1f06e3f829db5e2: Deleting consensus metadata
I20260928 12:33:27.389178 21974 catalog_manager.cc:4841] TS 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421): tablet 00c5f4ece9b34bba910a828880252d0f (table test-table [id=e073cedae0d9404ca98f323aabada83a]) successfully deleted
W20260928 12:33:27.392547 21972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:39173 (error 108)
I20260928 12:33:27.392887 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 22184
I20260928 12:33:27.393080 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.392962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39173 (local address 127.0.0.1:57578)
0928 12:33:27.393012 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.393046 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:33:27.393218 21972 catalog_manager.cc:4584] TS 15322d840f904ab8b3d71afe43387c80 (127.8.13.1:39173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00c5f4ece9b34bba910a828880252d0f: Network error: Client connection negotiation failed: client connection to 127.8.13.1:39173: connect: Connection refused (error 111)
I20260928 12:33:27.398074 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 22318
I20260928 12:33:27.402590 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 21956
I20260928 12:33:27.407133 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:34429
--webserver_interface=127.8.13.62
--webserver_port=43919
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:34429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:27.486985 22491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:27.487215 22491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:27.487248 22491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:27.488461 22491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:33:27.488509 22491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:27.488528 22491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:27.488543 22491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:33:27.488559 22491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:33:27.490193 22491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:34429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:34429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:27.490483 22491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:27.490697 22491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:27.499178 22497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:27.499193 22498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:27.499243 22491 server_base.cc:1034] running on GCE node
W20260928 12:33:27.499197 22500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:27.499771 22491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:27.500080 22491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:27.501214 22491 hybrid_clock.cc:648] HybridClock initialized: now 1790598807501201 us; error 29 us; skew 500 ppm
Sep 28 12:33:27 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598807, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:27.503747 22491 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:33:27.504227 22491 webserver.cc:460] Webserver started at http://127.8.13.62:43919/ using document root <none> and password file <none>
I20260928 12:33:27.504456 22491 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:27.504518 22491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:27.505879 22491 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:33:27.506495 22507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.506640 22491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:33:27.506721 22491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "305f40b26cf5e29dd46780e0c9e59c34"
server_key_iv: "fe15013719df15a861971cb4a9213131"
server_key_version: "encryptionkey@0"
I20260928 12:33:27.507123 22491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:27.517854 22491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:27.519600 22491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:27.519722 22491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:27.523424 22491 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:34429
I20260928 12:33:27.523455 22559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:34429 every 8 connection(s)
I20260928 12:33:27.523808 22491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:33:27.525271 22560 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:33:27.525926 22560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Bootstrap starting.
I20260928 12:33:27.531448 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 22491
I20260928 12:33:27.531700 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:39173
--local_ip_for_outbound_sockets=127.8.13.1
--tserver_master_addrs=127.8.13.62:34429
--webserver_port=33547
--webserver_interface=127.8.13.1
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:33:27.533707 22560 log.cc:824] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:27.535568 22560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:33:27.535815 22560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Bootstrap complete.
I20260928 12:33:27.538003 22560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:27.538247 22560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf67fd34ea14838a7ff9da02ffa1d3d, State: Initialized, Role: FOLLOWER
I20260928 12:33:27.538416 22560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:27.538499 22560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:33:27.538554 22560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:33:27.538600 22560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:33:27.539639 22560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:27.539742 22560 leader_election.cc:302] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cf67fd34ea14838a7ff9da02ffa1d3d; no voters:
I20260928 12:33:27.539909 22560 leader_election.cc:288] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:33:27.539965 22564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:33:27.540105 22564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 2 LEADER]: Becoming Leader. State: Replica: 9cf67fd34ea14838a7ff9da02ffa1d3d, State: Running, Role: LEADER
I20260928 12:33:27.540196 22560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:33:27.540257 22564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:27.540578 22565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } } }
I20260928 12:33:27.540656 22565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:27.540921 22566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cf67fd34ea14838a7ff9da02ffa1d3d. Latest consensus state: current_term: 2 leader_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } } }
I20260928 12:33:27.541002 22566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:27.541100 22569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:33:27.541934 22569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:33:27.542119 22569 catalog_manager.cc:1259] Loaded cluster ID: b16abdb8938b440ca4bb41c2db49113f
I20260928 12:33:27.542284 22569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:33:27.542520 22569 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 12:33:27.545289 22569 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 12:33:27.547175 22569 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:33:27.547358 22569 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Loaded TSK: 0
I20260928 12:33:27.547549 22569 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:33:27.632268 22562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:27.632480 22562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:27.632503 22562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:27.633639 22562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:27.633688 22562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:27.633714 22562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:33:27.635191 22562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:39173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=33547
--tserver_master_addrs=127.8.13.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:27.635412 22562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:27.635637 22562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:27.640764 22587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:27.640729 22590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:27.640756 22588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:27.641000 22562 server_base.cc:1034] running on GCE node
I20260928 12:33:27.641290 22562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:27.641606 22562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:27.642800 22562 hybrid_clock.cc:648] HybridClock initialized: now 1790598807642772 us; error 43 us; skew 500 ppm
Sep 28 12:33:27 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598807, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:27.648942 22562 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:33:27.649504 22562 webserver.cc:460] Webserver started at http://127.8.13.1:33547/ using document root <none> and password file <none>
I20260928 12:33:27.649722 22562 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:27.649783 22562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:27.651245 22562 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:27.652017 22597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.652237 22562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:27.652343 22562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "15322d840f904ab8b3d71afe43387c80"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "a237748e221ce9a6bdadb7ecd62de345"
server_key_iv: "8a27d713dc435705e2f5b4b416688942"
server_key_version: "encryptionkey@0"
I20260928 12:33:27.652724 22562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:27.672238 22562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:27.674505 22562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:27.674650 22562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:27.674867 22562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:27.675313 22604 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:33:27.677323 22562 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:33:27.677420 22562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:33:27.677459 22562 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:33:27.678185 22562 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:33:27.678221 22562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:33:27.678310 22604 tablet_bootstrap.cc:492] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Bootstrap starting.
I20260928 12:33:27.685065 22562 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:39173
I20260928 12:33:27.685153 22711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:39173 every 8 connection(s)
I20260928 12:33:27.685547 22562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:33:27.686923 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 22562
I20260928 12:33:27.687206 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:37421
--local_ip_for_outbound_sockets=127.8.13.2
--tserver_master_addrs=127.8.13.62:34429
--webserver_port=39417
--webserver_interface=127.8.13.2
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:33:27.689828 22604 log.cc:824] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Log is configured to *not* fsync() on all Append() calls
Sep 28 12:33:27 dist-test-slave-g31s krb5kdc[21943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598807, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:27.698454 22604 tablet_bootstrap.cc:492] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:33:27.698824 22604 tablet_bootstrap.cc:492] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Bootstrap complete.
I20260928 12:33:27.699956 22604 ts_tablet_manager.cc:1397] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.009s
I20260928 12:33:27.701282 22604 raft_consensus.cc:348] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.701620 22604 raft_consensus.cc:729] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15322d840f904ab8b3d71afe43387c80, State: Initialized, Role: FOLLOWER
I20260928 12:33:27.701786 22604 consensus_queue.cc:260] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.701989 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.686318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:60223)
0928 12:33:27.686514 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.686566 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.687176 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.690516 (+ 3340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.690522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.690913 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:27.691229 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.691238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.692889 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.692891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.694185 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.694189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.695050 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.695051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.695196 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.696013 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:27.696023 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:27.697938 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:27.700108 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:27.700115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:27.700117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:27.700394 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:27.700738 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:27.700738 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:27.700739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:27.700864 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:27.701308 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:27.701314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:27.701433 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.701529 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.701710 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260928 12:33:27.702131 22604 ts_tablet_manager.cc:1428] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:33:27.702414 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.686397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:60223 (local address 127.8.13.62:34429)
0928 12:33:27.689965 (+ 3568us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.689977 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.690030 (+ 53us) server_negotiation.cc:407] Connection header received
0928 12:33:27.690176 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.690183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.690250 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.690455 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:27.691369 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.692748 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.694283 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.694935 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.695068 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.698071 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:27.698076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:27.698077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:27.698089 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:27.699978 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.700553 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.700556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.700557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.700672 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.700931 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.700932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.700932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.701226 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:27.701304 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.701593 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.701888 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3505,"thread_start_us":70,"threads_started":1}
I20260928 12:33:27.702917 22712 heartbeater.cc:347] Connected to a master server at 127.8.13.62:34429
I20260928 12:33:27.703006 22712 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:27.703167 22712 heartbeater.cc:511] Master 127.8.13.62:34429 requested a full tablet report, sending...
I20260928 12:33:27.703773 22524 ts_manager.cc:194] Registered new tserver with Master: 15322d840f904ab8b3d71afe43387c80 (127.8.13.1:39173)
I20260928 12:33:27.704734 22524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:60223
I20260928 12:33:27.705603 22712 heartbeater.cc:503] Master 127.8.13.62:34429 was elected leader, sending a full tablet report...
Sep 28 12:33:27 dist-test-slave-g31s krb5kdc[21943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598807, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:33:27.712842 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.704311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39173 (local address 127.0.0.1:57592)
0928 12:33:27.704478 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.704520 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.704586 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.705279 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.705284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.705597 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:27.705658 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.705660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.707001 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.707001 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.707555 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.707557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.708146 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.708149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.708196 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.708651 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:27.708655 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:27.710098 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:27.711926 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:27.711932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:27.711933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:27.712113 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:27.712291 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:27.712292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:27.712292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:27.712335 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:27.712595 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:27.712599 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:27.712638 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.712692 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.712775 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260928 12:33:27.712925 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.704992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.8.13.1:39173)
0928 12:33:27.705138 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.705142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.705151 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:33:27.705181 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.705183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.705199 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.705268 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:27.705747 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.706901 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.707626 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.708084 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.708151 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.710177 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:27.710185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:27.710185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:27.710192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:27.711841 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.712177 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.712180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.712180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.712239 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.712380 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.712381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.712381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.712537 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:27.712605 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.712736 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.712827 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260928 12:33:27.713261 22646 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c5f4ece9b34bba910a828880252d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:27 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:57592
I20260928 12:33:27.713850 22721 tablet_replica.cc:330] stopping tablet replica
I20260928 12:33:27.713953 22721 raft_consensus.cc:2229] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:33:27.714015 22721 raft_consensus.cc:2258] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:33:27.714278 22721 ts_tablet_manager.cc:1905] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:33:27.716070 22721 ts_tablet_manager.cc:1918] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:33:27.716133 22721 log.cc:1196] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal/wals/00c5f4ece9b34bba910a828880252d0f
I20260928 12:33:27.716326 22721 ts_tablet_manager.cc:1939] T 00c5f4ece9b34bba910a828880252d0f P 15322d840f904ab8b3d71afe43387c80: Deleting consensus metadata
I20260928 12:33:27.716734 22508 catalog_manager.cc:4841] TS 15322d840f904ab8b3d71afe43387c80 (127.8.13.1:39173): tablet 00c5f4ece9b34bba910a828880252d0f (table test-table [id=e073cedae0d9404ca98f323aabada83a]) successfully deleted
W20260928 12:33:27.774518 22716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:27.774744 22716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:27.774770 22716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:27.776068 22716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:27.776129 22716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:27.776158 22716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:33:27.777881 22716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:37421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=39417
--tserver_master_addrs=127.8.13.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:27.778131 22716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:27.778349 22716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:27.783408 22729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:27.783413 22726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:27.783597 22727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:27.783680 22716 server_base.cc:1034] running on GCE node
I20260928 12:33:27.783895 22716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:27.784152 22716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:27.785297 22716 hybrid_clock.cc:648] HybridClock initialized: now 1790598807785284 us; error 29 us; skew 500 ppm
Sep 28 12:33:27 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598807, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:27.787775 22716 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:33:27.788223 22716 webserver.cc:460] Webserver started at http://127.8.13.2:39417/ using document root <none> and password file <none>
I20260928 12:33:27.788406 22716 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:27.788458 22716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:27.789786 22716 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:27.790364 22736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.790521 22716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.790603 22716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "629d0005d3e74139a1f06e3f829db5e2"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "5d9ac410fdcd5c02088fe94b444da3ca"
server_key_iv: "7b78b69a82e211c3d92a103d3f8ad619"
server_key_version: "encryptionkey@0"
I20260928 12:33:27.790941 22716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:27.802743 22716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:27.804772 22716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:27.804878 22716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:27.805058 22716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:27.805297 22716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:27.805325 22716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.805347 22716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:27.805363 22716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.810708 22716 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:37421
I20260928 12:33:27.810771 22849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:37421 every 8 connection(s)
I20260928 12:33:27.811057 22716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:33:27.812067 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 22716
I20260928 12:33:27.812295 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:37479
--local_ip_for_outbound_sockets=127.8.13.3
--tserver_master_addrs=127.8.13.62:34429
--webserver_port=33099
--webserver_interface=127.8.13.3
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:33:27 dist-test-slave-g31s krb5kdc[21943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598807, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:27.819742 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.811679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:43309 (local address 127.8.13.62:34429)
0928 12:33:27.811734 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.811736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.811925 (+ 189us) server_negotiation.cc:407] Connection header received
0928 12:33:27.812489 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.812490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.812502 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.812564 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:27.813458 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.814025 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.814965 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.815182 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.815236 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.817217 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:27.817218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:27.817218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:27.817226 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:27.818463 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.818821 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.818823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.818823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.818876 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.819091 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.819092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.819092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.819200 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:27.819255 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.819548 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.819641 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:33:27.819847 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.811670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:43309)
0928 12:33:27.811817 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.811852 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.812391 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.812634 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.812638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.813185 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:27.813367 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.813372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.814105 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.814106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.814887 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.814889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.815251 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.815252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.815378 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.815855 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:27.815863 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:27.817135 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:27.818541 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:27.818547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:27.818549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:27.818759 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:27.818931 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:27.818931 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:27.818932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:27.819037 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:27.819253 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:27.819260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:27.819393 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.819490 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.819656 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260928 12:33:27.820284 22850 heartbeater.cc:347] Connected to a master server at 127.8.13.62:34429
I20260928 12:33:27.820371 22850 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:27.820544 22850 heartbeater.cc:511] Master 127.8.13.62:34429 requested a full tablet report, sending...
I20260928 12:33:27.820897 22524 ts_manager.cc:194] Registered new tserver with Master: 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421)
I20260928 12:33:27.821394 22524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:43309
W20260928 12:33:27.894236 22853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:27.894503 22853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:27.894542 22853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:27.895750 22853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:27.895807 22853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:27.895845 22853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:33:27.897357 22853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:37479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=33099
--tserver_master_addrs=127.8.13.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:27.897622 22853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:27.897886 22853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:27.903025 22858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:27.903112 22853 server_base.cc:1034] running on GCE node
W20260928 12:33:27.903060 22861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:27.903079 22859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:27.903533 22853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:27.903801 22853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:27.904954 22853 hybrid_clock.cc:648] HybridClock initialized: now 1790598807904933 us; error 36 us; skew 500 ppm
Sep 28 12:33:27 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598807, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:27.907335 22853 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:33:27.907773 22853 webserver.cc:460] Webserver started at http://127.8.13.3:33099/ using document root <none> and password file <none>
I20260928 12:33:27.907949 22853 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:27.907999 22853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:27.909237 22853 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:33:27.909818 22868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.909951 22853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.910027 22853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "f158271212e04ec58df6d46ef2bbbd62"
format_stamp: "Formatted at 2026-09-28 12:33:27 on dist-test-slave-g31s"
server_key: "e021115452ea73dd1d5a1c48c2f1203c"
server_key_iv: "9b7b9f84992cfc59ea6cba9d093cadf2"
server_key_version: "encryptionkey@0"
I20260928 12:33:27.910416 22853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:27.928704 22853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:27.930759 22853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:27.930879 22853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:27.931104 22853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:27.931407 22853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:27.931483 22853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.931559 22853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:27.931586 22853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.937239 22853 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:37479
I20260928 12:33:27.937292 22981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:37479 every 8 connection(s)
I20260928 12:33:27.937641 22853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:33:27 dist-test-slave-g31s krb5kdc[21943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598807, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:27.946928 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.938327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:47571 (local address 127.8.13.62:34429)
0928 12:33:27.938375 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.938377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.938586 (+ 209us) server_negotiation.cc:407] Connection header received
0928 12:33:27.939116 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.939117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.939131 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.939181 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:27.940141 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.940689 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.941673 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.941926 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.941984 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.944220 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:27.944222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:27.944222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:27.944230 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:27.945344 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.945788 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.945789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.945789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.945899 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.946212 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.946214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.946214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.946357 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:27.946424 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.946757 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.946839 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:33:27.947048 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.938330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:47571)
0928 12:33:27.938462 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.938501 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.939011 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.939251 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.939254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.939815 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:27.940022 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.940026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.940779 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.940780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.941582 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.941584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.942009 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.942010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.942133 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.942657 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:27.942666 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:27.944124 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:27.945467 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:27.945475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:27.945477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:27.945725 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:27.945969 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:27.945970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:27.945970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:27.946123 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:27.946440 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:27.946448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:27.946595 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.946687 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.946831 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260928 12:33:27.947252 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 22853
I20260928 12:33:27.947682 22982 heartbeater.cc:347] Connected to a master server at 127.8.13.62:34429
I20260928 12:33:27.947799 22982 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:27.948022 22982 heartbeater.cc:511] Master 127.8.13.62:34429 requested a full tablet report, sending...
I20260928 12:33:27.948469 22524 ts_manager.cc:194] Registered new tserver with Master: f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479)
I20260928 12:33:27.948966 22524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:47571
I20260928 12:33:27.953387 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.947574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.8.13.62:34429)
0928 12:33:27.947624 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.947626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.947729 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:33:27.947789 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.947790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.947801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.947852 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:27.948502 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.949228 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.950059 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.950251 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.950305 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.951465 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:27.951467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:27.951467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:27.951473 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:27.952581 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.952864 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.952865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.952865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.952921 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:27.953028 (+ 107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:27.953028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:27.953029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:27.953122 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:27.953238 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.953263 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.953319 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:33:27.954139 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:33:27.956785 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.954464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.8.13.62:34429)
0928 12:33:27.954514 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.954515 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.954553 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:33:27.954622 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.954623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.954663 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.954724 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:33:27.954833 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.955313 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.956109 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.956296 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.956346 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.956509 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:33:27.956613 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:33:27.956681 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.956701 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.956718 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:33:27.957085 22524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:33:27.958789 22524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:33:27.961010 22646 tablet_service.cc:1467] Processing CreateTablet for tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53 (DEFAULT_TABLE table=test-table [id=ce7f2b915fa74eeab2cfab46a3d791b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:27.961218 22646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:27.962798 22604 tablet_bootstrap.cc:492] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: Bootstrap starting.
I20260928 12:33:27.963119 22604 tablet_bootstrap.cc:654] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:27.963835 22604 tablet_bootstrap.cc:492] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: No bootstrap required, opened a new log
I20260928 12:33:27.963881 22604 ts_tablet_manager.cc:1397] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:33:27.964031 22604 raft_consensus.cc:348] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.964090 22604 raft_consensus.cc:374] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:27.964143 22604 raft_consensus.cc:729] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15322d840f904ab8b3d71afe43387c80, State: Initialized, Role: FOLLOWER
I20260928 12:33:27.964196 22604 consensus_queue.cc:260] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.964288 22604 ts_tablet_manager.cc:1428] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:27.964671 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.960613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:37479 (local address 127.0.0.1:32926)
0928 12:33:27.960751 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.960761 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.960815 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.961240 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.961241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.961243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:27.961290 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.961291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.962283 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.962284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.963517 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.963519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.964412 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.964413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.964489 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.964518 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.964567 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.964613 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":36,"thread_start_us":85,"threads_started":1}
I20260928 12:33:27.964968 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.960918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.8.13.3:37479)
0928 12:33:27.961066 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.961070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.961078 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:33:27.961112 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.961114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.961129 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.961234 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:27.961350 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.962189 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.963646 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.964326 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.964395 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.964475 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.964838 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.964900 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260928 12:33:27.965596 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.960613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:37421 (local address 127.0.0.1:49292)
0928 12:33:27.960665 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:27.960675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:27.960710 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:27.961131 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:27.961131 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:27.961133 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:27.961189 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.961190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.962511 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.962512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.963563 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:27.963565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.965313 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.965314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:27.965425 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.965430 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:33:27.965483 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:33:27.965535 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:33:27.965617 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:27.960730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.8.13.2:37421)
0928 12:33:27.960927 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:27.960932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:27.960941 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:33:27.960979 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:27.960981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:27.960996 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:27.961114 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:27.961378 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.962411 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.964514 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:27.965220 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:27.965284 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:27.965339 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:27.965508 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 12:33:27.965550 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
I20260928 12:33:27.965763 22916 tablet_service.cc:1467] Processing CreateTablet for tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53 (DEFAULT_TABLE table=test-table [id=ce7f2b915fa74eeab2cfab46a3d791b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:27.966169 22916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:27.966265 22784 tablet_service.cc:1467] Processing CreateTablet for tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53 (DEFAULT_TABLE table=test-table [id=ce7f2b915fa74eeab2cfab46a3d791b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:27.966583 22784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:27.969962 22991 tablet_bootstrap.cc:492] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: Bootstrap starting.
I20260928 12:33:27.969962 22992 tablet_bootstrap.cc:492] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: Bootstrap starting.
I20260928 12:33:27.970703 22991 tablet_bootstrap.cc:654] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:27.970703 22992 tablet_bootstrap.cc:654] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:27.971293 22991 log.cc:824] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:27.971293 22992 log.cc:824] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:27.972134 22991 tablet_bootstrap.cc:492] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: No bootstrap required, opened a new log
I20260928 12:33:27.972134 22992 tablet_bootstrap.cc:492] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: No bootstrap required, opened a new log
I20260928 12:33:27.972208 22992 ts_tablet_manager.cc:1397] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:33:27.972223 22991 ts_tablet_manager.cc:1397] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:33:27.974225 22992 raft_consensus.cc:348] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.974225 22991 raft_consensus.cc:348] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.974433 22991 raft_consensus.cc:374] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:27.974433 22992 raft_consensus.cc:374] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:27.974471 22991 raft_consensus.cc:729] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629d0005d3e74139a1f06e3f829db5e2, State: Initialized, Role: FOLLOWER
I20260928 12:33:27.974488 22992 raft_consensus.cc:729] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f158271212e04ec58df6d46ef2bbbd62, State: Initialized, Role: FOLLOWER
I20260928 12:33:27.974601 22991 consensus_queue.cc:260] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.974601 22992 consensus_queue.cc:260] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:27.974902 22992 ts_tablet_manager.cc:1428] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 12:33:27.974902 22991 ts_tablet_manager.cc:1428] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:33:27.974925 22982 heartbeater.cc:503] Master 127.8.13.62:34429 was elected leader, sending a full tablet report...
I20260928 12:33:27.975018 22850 heartbeater.cc:503] Master 127.8.13.62:34429 was elected leader, sending a full tablet report...
W20260928 12:33:28.061590 22851 tablet.cc:2369] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:33:28.186488 22713 tablet.cc:2369] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:33:28.188162 22983 tablet.cc:2369] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:33:28.291100 22995 raft_consensus.cc:482] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:33:28.291226 22995 raft_consensus.cc:504] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:28.291591 22995 leader_election.cc:288] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421), 15322d840f904ab8b3d71afe43387c80 (127.8.13.1:39173)
I20260928 12:33:28.294524 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.291852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:32975 (local address 127.8.13.1:39173)
0928 12:33:28.292006 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:28.292007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:28.292016 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:33:28.292046 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:28.292047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:28.292075 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:28.292144 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:28.292277 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:28.292768 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:28.293726 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:28.294191 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:28.294250 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:28.294327 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:28.294413 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 12:33:28.294451 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260928 12:33:28.294529 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.291787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39173 (local address 127.8.13.3:32975)
0928 12:33:28.291833 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.291844 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:28.291891 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:28.292154 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:28.292155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:28.292157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:28.292221 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:28.292223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:28.292835 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:28.292836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:28.293645 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:28.293648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:28.294262 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:28.294262 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:28.294340 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:28.294344 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:33:28.294406 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:33:28.294449 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 12:33:28.294549 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.291789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:37421 (local address 127.8.13.3:39801)
0928 12:33:28.292097 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.292105 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:28.292163 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:28.292293 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:28.292293 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:28.292295 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:28.292361 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:28.292362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:28.292954 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:28.292955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:28.293724 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:28.293725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:28.294294 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:28.294295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:28.294365 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:28.294370 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:33:28.294424 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:33:28.294460 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":149,"thread_start_us":93,"threads_started":1}
I20260928 12:33:28.294541 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.291835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:39801 (local address 127.8.13.2:37421)
0928 12:33:28.291889 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:28.291891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:28.292164 (+ 273us) server_negotiation.cc:407] Connection header received
0928 12:33:28.292223 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:28.292224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:28.292236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:28.292283 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:28.292439 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:28.292878 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:28.293825 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:28.294224 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:28.294285 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:28.294294 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:28.294430 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 12:33:28.294462 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:33:28.294965 22666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df3ba1d9cca4445a46f9e5dc7f7cb53" candidate_uuid: "f158271212e04ec58df6d46ef2bbbd62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15322d840f904ab8b3d71afe43387c80" is_pre_election: true
I20260928 12:33:28.294975 22804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df3ba1d9cca4445a46f9e5dc7f7cb53" candidate_uuid: "f158271212e04ec58df6d46ef2bbbd62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629d0005d3e74139a1f06e3f829db5e2" is_pre_election: true
I20260928 12:33:28.295114 22804 raft_consensus.cc:2454] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f158271212e04ec58df6d46ef2bbbd62 in term 0.
I20260928 12:33:28.295111 22666 raft_consensus.cc:2454] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f158271212e04ec58df6d46ef2bbbd62 in term 0.
I20260928 12:33:28.295327 22871 leader_election.cc:302] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 629d0005d3e74139a1f06e3f829db5e2, f158271212e04ec58df6d46ef2bbbd62; no voters:
I20260928 12:33:28.295449 22995 raft_consensus.cc:2781] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:33:28.295502 22995 raft_consensus.cc:482] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:33:28.295524 22995 raft_consensus.cc:3037] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:28.296198 22995 raft_consensus.cc:504] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:28.296317 22995 leader_election.cc:288] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [CANDIDATE]: Term 1 election: Requested vote from peers 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421), 15322d840f904ab8b3d71afe43387c80 (127.8.13.1:39173)
I20260928 12:33:28.296512 22666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df3ba1d9cca4445a46f9e5dc7f7cb53" candidate_uuid: "f158271212e04ec58df6d46ef2bbbd62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15322d840f904ab8b3d71afe43387c80"
I20260928 12:33:28.296530 22804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df3ba1d9cca4445a46f9e5dc7f7cb53" candidate_uuid: "f158271212e04ec58df6d46ef2bbbd62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629d0005d3e74139a1f06e3f829db5e2"
I20260928 12:33:28.296591 22666 raft_consensus.cc:3037] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:28.296594 22804 raft_consensus.cc:3037] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:28.297230 22804 raft_consensus.cc:2454] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f158271212e04ec58df6d46ef2bbbd62 in term 1.
I20260928 12:33:28.297240 22666 raft_consensus.cc:2454] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f158271212e04ec58df6d46ef2bbbd62 in term 1.
I20260928 12:33:28.297405 22869 leader_election.cc:302] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15322d840f904ab8b3d71afe43387c80, f158271212e04ec58df6d46ef2bbbd62; no voters:
I20260928 12:33:28.297506 22995 raft_consensus.cc:2781] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:33:28.297653 22995 raft_consensus.cc:686] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 1 LEADER]: Becoming Leader. State: Replica: f158271212e04ec58df6d46ef2bbbd62, State: Running, Role: LEADER
I20260928 12:33:28.297739 22995 consensus_queue.cc:237] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:28.298558 22522 catalog_manager.cc:5495] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 reported cstate change: term changed from 0 to 1, leader changed from <none> to f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3). New cstate: current_term: 1 leader_uuid: "f158271212e04ec58df6d46ef2bbbd62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } health_report { overall_health: UNKNOWN } } }
I20260928 12:33:28.323076 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.321040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.8.13.3:37479)
0928 12:33:28.321090 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:28.321091 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:28.321102 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:33:28.321174 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:28.321175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:28.321194 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:28.321252 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:33:28.321334 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:28.321790 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:28.322493 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:28.322670 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:28.322716 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:28.322808 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:33:28.322899 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:33:28.322949 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:28.322964 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 12:33:28.322979 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:33:28.324708 22804 raft_consensus.cc:1261] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2 [term 1 FOLLOWER]: Refusing update from remote peer f158271212e04ec58df6d46ef2bbbd62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:33:28.324745 22666 raft_consensus.cc:1261] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Refusing update from remote peer f158271212e04ec58df6d46ef2bbbd62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:33:28.325054 22995 consensus_queue.cc:1035] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:33:28.325179 22999 consensus_queue.cc:1035] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:33:28.327414 23003 mvcc.cc:205] Tried to move back new op lower bound from 7334292718895480832 to 7334292718788182016. Current Snapshot: MvccSnapshot[applied={T|T < 7334292718895480832}]
I20260928 12:33:28.327913 23004 mvcc.cc:205] Tried to move back new op lower bound from 7334292718895480832 to 7334292718788182016. Current Snapshot: MvccSnapshot[applied={T|T < 7334292718895480832}]
I20260928 12:33:28.329772 22522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:33:28.329896 22522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:33:28.330536 22522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Sending DeleteTablet for 3 replicas of tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53
I20260928 12:33:28.330825 22646 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:28 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:57592
I20260928 12:33:28.330832 22784 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:28 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:49292
I20260928 12:33:28.330892 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 22562
I20260928 12:33:28.331313 22916 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:28 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:32926
I20260928 12:33:28.331794 23014 tablet_replica.cc:330] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: stopping tablet replica
I20260928 12:33:28.331903 23014 raft_consensus.cc:2229] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:33:28.332019 23014 raft_consensus.cc:2258] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:33:28.332160 23013 tablet_replica.cc:330] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: stopping tablet replica
I20260928 12:33:28.332244 23013 raft_consensus.cc:2229] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:33:28.332330 23013 raft_consensus.cc:2258] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:33:28.332670 23014 ts_tablet_manager.cc:1905] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:33:28.334581 23013 ts_tablet_manager.cc:1905] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:33:28.336009 23013 ts_tablet_manager.cc:1918] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:33:28.336084 23013 log.cc:1196] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/2df3ba1d9cca4445a46f9e5dc7f7cb53
I20260928 12:33:28.336199 23014 ts_tablet_manager.cc:1918] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:33:28.336265 23014 log.cc:1196] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/2df3ba1d9cca4445a46f9e5dc7f7cb53
I20260928 12:33:28.336544 23014 ts_tablet_manager.cc:1939] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P f158271212e04ec58df6d46ef2bbbd62: Deleting consensus metadata
I20260928 12:33:28.337052 22508 catalog_manager.cc:4841] TS f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479): tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53 (table test-table [id=ce7f2b915fa74eeab2cfab46a3d791b6]) successfully deleted
I20260928 12:33:28.337129 23013 ts_tablet_manager.cc:1939] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 629d0005d3e74139a1f06e3f829db5e2: Deleting consensus metadata
I20260928 12:33:28.337553 22510 catalog_manager.cc:4841] TS 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421): tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53 (table test-table [id=ce7f2b915fa74eeab2cfab46a3d791b6]) successfully deleted
W20260928 12:33:28.337883 22508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:39173 (error 108)
I20260928 12:33:28.338145 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 22716
I20260928 12:33:28.338403 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.338273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:39173 (local address 127.0.0.1:57602)
0928 12:33:28.338327 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.338359 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:33:28.340435 22508 catalog_manager.cc:4584] TS 15322d840f904ab8b3d71afe43387c80 (127.8.13.1:39173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2df3ba1d9cca4445a46f9e5dc7f7cb53: Network error: Client connection negotiation failed: client connection to 127.8.13.1:39173: connect: Connection refused (error 111)
I20260928 12:33:28.343350 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 22853
I20260928 12:33:28.348569 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 22491
I20260928 12:33:28.353220 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:34429
--webserver_interface=127.8.13.62
--webserver_port=43919
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:34429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:28.432832 23016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:28.433044 23016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:28.433065 23016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:28.434279 23016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:33:28.434327 23016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:28.434340 23016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:28.434352 23016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:33:28.434372 23016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:33:28.435932 23016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:34429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:34429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:28.436210 23016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:28.436486 23016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:28.441388 23021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:28.441375 23024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:28.441531 23016 server_base.cc:1034] running on GCE node
W20260928 12:33:28.441380 23022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:28.441926 23016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:28.442247 23016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:28.443394 23016 hybrid_clock.cc:648] HybridClock initialized: now 1790598808443379 us; error 31 us; skew 500 ppm
Sep 28 12:33:28 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:28.445876 23016 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:33:28.446305 23016 webserver.cc:460] Webserver started at http://127.8.13.62:43919/ using document root <none> and password file <none>
I20260928 12:33:28.446511 23016 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:28.446559 23016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:28.447969 23016 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:28.448596 23031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:28.448740 23016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:33:28.448823 23016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "305f40b26cf5e29dd46780e0c9e59c34"
server_key_iv: "fe15013719df15a861971cb4a9213131"
server_key_version: "encryptionkey@0"
I20260928 12:33:28.449173 23016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:28.471560 23016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:28.473974 23016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:28.474095 23016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:28.478073 23016 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:34429
I20260928 12:33:28.478109 23083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:34429 every 8 connection(s)
I20260928 12:33:28.478538 23016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:33:28.480130 23084 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:33:28.480782 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Bootstrap starting.
I20260928 12:33:28.488144 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 23016
I20260928 12:33:28.488405 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:39173
--local_ip_for_outbound_sockets=127.8.13.1
--tserver_master_addrs=127.8.13.62:34429
--webserver_port=33547
--webserver_interface=127.8.13.1
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:33:28.488967 23084 log.cc:824] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:28.491678 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:33:28.491971 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d: Bootstrap complete.
I20260928 12:33:28.494155 23084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:28.494385 23084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf67fd34ea14838a7ff9da02ffa1d3d, State: Initialized, Role: FOLLOWER
I20260928 12:33:28.494534 23084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:28.494609 23084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:33:28.494660 23084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:33:28.494706 23084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 2 FOLLOWER]: Advancing to term 3
I20260928 12:33:28.495712 23084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:28.495807 23084 leader_election.cc:302] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cf67fd34ea14838a7ff9da02ffa1d3d; no voters:
I20260928 12:33:28.495987 23084 leader_election.cc:288] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 12:33:28.496037 23088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 3 FOLLOWER]: Leader election won for term 3
I20260928 12:33:28.496199 23088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [term 3 LEADER]: Becoming Leader. State: Replica: 9cf67fd34ea14838a7ff9da02ffa1d3d, State: Running, Role: LEADER
I20260928 12:33:28.496245 23084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:33:28.496366 23088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } }
I20260928 12:33:28.496681 23090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cf67fd34ea14838a7ff9da02ffa1d3d. Latest consensus state: current_term: 3 leader_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } } }
I20260928 12:33:28.496688 23089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf67fd34ea14838a7ff9da02ffa1d3d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34429 } } }
I20260928 12:33:28.496779 23090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:28.496786 23089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf67fd34ea14838a7ff9da02ffa1d3d [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:28.496990 23097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:33:28.497897 23097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:33:28.498028 23097 catalog_manager.cc:1259] Loaded cluster ID: b16abdb8938b440ca4bb41c2db49113f
I20260928 12:33:28.498267 23097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 12:33:28.498637 23097 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f14158d09ad google::LogMessage::Fail() at ??:0
@ 0x7f14158d388a google::LogMessage::SendToLog() at ??:0
@ 0x7f14158d04de google::LogMessage::Flush() at ??:0
@ 0x7f14158d1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f141d7f10f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f141d7fa8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f141779e0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1417793954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f14185f76db start_thread at ??:0
@ 0x7f14143ce61f clone at ??:0
W20260928 12:33:28.581459 23087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:28.581684 23087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:28.581712 23087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:28.583050 23087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:28.583109 23087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:28.583139 23087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:33:28.584784 23087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:39173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=33547
--tserver_master_addrs=127.8.13.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:28.585060 23087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:28.585300 23087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:28.591444 23121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:28.591425 23120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:28.591420 23123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:28.591745 23087 server_base.cc:1034] running on GCE node
I20260928 12:33:28.591955 23087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:28.592206 23087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:28.593390 23087 hybrid_clock.cc:648] HybridClock initialized: now 1790598808593373 us; error 32 us; skew 500 ppm
Sep 28 12:33:28 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:28.596143 23087 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:33:28.596655 23087 webserver.cc:460] Webserver started at http://127.8.13.1:33547/ using document root <none> and password file <none>
I20260928 12:33:28.596891 23087 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:28.598111 23087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:28.599987 23087 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:33:28.600879 23130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:28.601056 23087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:33:28.601156 23087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "15322d840f904ab8b3d71afe43387c80"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "a237748e221ce9a6bdadb7ecd62de345"
server_key_iv: "8a27d713dc435705e2f5b4b416688942"
server_key_version: "encryptionkey@0"
I20260928 12:33:28.601610 23087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:28.629191 23087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:28.631687 23087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:28.631841 23087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:28.632105 23087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:28.632607 23137 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:33:28.634104 23087 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:33:28.634166 23087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:33:28.634202 23087 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:33:28.635149 23087 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:33:28.635196 23087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:33:28.635244 23137 tablet_bootstrap.cc:492] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: Bootstrap starting.
I20260928 12:33:28.642443 23087 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:39173
I20260928 12:33:28.642546 23244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:39173 every 8 connection(s)
I20260928 12:33:28.643064 23087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:33:28.643994 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 23087
I20260928 12:33:28.644316 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:37421
--local_ip_for_outbound_sockets=127.8.13.2
--tserver_master_addrs=127.8.13.62:34429
--webserver_port=39417
--webserver_interface=127.8.13.2
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:33:28.645520 23137 log.cc:824] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:28.647895 23137 tablet_bootstrap.cc:492] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:33:28.648192 23137 tablet_bootstrap.cc:492] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: Bootstrap complete.
I20260928 12:33:28.649324 23137 ts_tablet_manager.cc:1397] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260928 12:33:28.650959 23137 raft_consensus.cc:348] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:28.651321 23137 raft_consensus.cc:729] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15322d840f904ab8b3d71afe43387c80, State: Initialized, Role: FOLLOWER
I20260928 12:33:28.651500 23137 consensus_queue.cc:260] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:28.651818 23137 ts_tablet_manager.cc:1428] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260928 12:33:28.727859 23248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:28.728066 23248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:28.728089 23248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:28.729404 23248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:28.729466 23248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:28.729506 23248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:33:28.731020 23248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:37421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=39417
--tserver_master_addrs=127.8.13.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:28.731243 23248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:28.731458 23248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:28.732653 23247 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:33:28.643814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:58961)
0928 12:33:28.644007 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.644051 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:28.644603 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:28.732446 (+ 87843us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":112,"threads_started":1}
W20260928 12:33:28.732884 23245 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:34429 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.62:34429: Client connection negotiation failed: client connection to 127.8.13.62:34429: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:33:28.733186 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.733079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:38317)
0928 12:33:28.733127 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.733165 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 12:33:28.733567 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.733432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:52565)
0928 12:33:28.733510 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.733543 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 12:33:28.733718 23245 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:33:28.736496 23255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:28.736523 23258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:28.736534 23256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:28.736665 23248 server_base.cc:1034] running on GCE node
I20260928 12:33:28.736934 23248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:28.737212 23248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:28.738364 23248 hybrid_clock.cc:648] HybridClock initialized: now 1790598808738347 us; error 28 us; skew 500 ppm
Sep 28 12:33:28 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:28.741040 23248 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:33:28.741489 23248 webserver.cc:460] Webserver started at http://127.8.13.2:39417/ using document root <none> and password file <none>
I20260928 12:33:28.741700 23248 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:28.741745 23248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:28.743144 23248 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:28.743757 23265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:28.743909 23248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:33:28.743991 23248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "629d0005d3e74139a1f06e3f829db5e2"
format_stamp: "Formatted at 2026-09-28 12:33:26 on dist-test-slave-g31s"
server_key: "5d9ac410fdcd5c02088fe94b444da3ca"
server_key_iv: "7b78b69a82e211c3d92a103d3f8ad619"
server_key_version: "encryptionkey@0"
I20260928 12:33:28.744330 23248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:28.758792 23248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:28.761071 23248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:28.761188 23248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:28.761385 23248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:28.761687 23248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:28.761719 23248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:28.761744 23248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:28.761766 23248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:28.767426 23248 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:37421
I20260928 12:33:28.767493 23378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:37421 every 8 connection(s)
I20260928 12:33:28.767814 23248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:33:28.768877 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.768520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:57277)
0928 12:33:28.768659 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.768731 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
W20260928 12:33:28.769233 23379 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:34429 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.62:34429: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
I20260928 12:33:28.769519 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.769402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:55123)
0928 12:33:28.769464 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.769496 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 12:33:28.769760 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.769696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:33329)
0928 12:33:28.769726 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.769747 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:33:28.769869 23379 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:33:28.770987 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 23248
I20260928 12:33:28.771226 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:37479
--local_ip_for_outbound_sockets=127.8.13.3
--tserver_master_addrs=127.8.13.62:34429
--webserver_port=33099
--webserver_interface=127.8.13.3
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:28.852494 23382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:28.852741 23382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:28.852767 23382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:28.854076 23382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:28.854139 23382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:28.854169 23382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:33:28.855859 23382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:37479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=33099
--tserver_master_addrs=127.8.13.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:28.856128 23382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:28.856376 23382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:28.861352 23388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:28.861352 23390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:28.861434 23382 server_base.cc:1034] running on GCE node
W20260928 12:33:28.861352 23387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:28.861766 23382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:28.862058 23382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:28.863216 23382 hybrid_clock.cc:648] HybridClock initialized: now 1790598808863202 us; error 31 us; skew 500 ppm
Sep 28 12:33:28 dist-test-slave-g31s krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:28.865904 23382 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:33:28.866364 23382 webserver.cc:460] Webserver started at http://127.8.13.3:33099/ using document root <none> and password file <none>
I20260928 12:33:28.866580 23382 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:28.866633 23382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:28.868080 23382 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:28.868664 23397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:28.868824 23382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:28.868906 23382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "f158271212e04ec58df6d46ef2bbbd62"
format_stamp: "Formatted at 2026-09-28 12:33:27 on dist-test-slave-g31s"
server_key: "e021115452ea73dd1d5a1c48c2f1203c"
server_key_iv: "9b7b9f84992cfc59ea6cba9d093cadf2"
server_key_version: "encryptionkey@0"
I20260928 12:33:28.869256 23382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:28.879652 23382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:28.881852 23382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:28.882033 23382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:28.882264 23382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:28.882589 23382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:28.882624 23382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:28.882655 23382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:28.882676 23382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:28.888686 23382 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:37479
I20260928 12:33:28.888722 23510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:37479 every 8 connection(s)
I20260928 12:33:28.889101 23382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:33:28.890185 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.889781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:53327)
0928 12:33:28.889966 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.890038 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
W20260928 12:33:28.890589 23511 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:34429 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.62:34429: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
I20260928 12:33:28.890865 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.890771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:59577)
0928 12:33:28.890807 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.890841 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 12:33:28.891170 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:28.891057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:46149)
0928 12:33:28.891117 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:28.891148 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 12:33:28.891278 23511 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:33:28.896206 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 23382
I20260928 12:33:29.734344 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:29.734104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:57429)
0928 12:33:29.734232 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:29.734274 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260928 12:33:29.770499 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:29.770256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:47101)
0928 12:33:29.770392 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:29.770441 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260928 12:33:29.891850 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:29.891632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:40229)
0928 12:33:29.891752 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:29.891793 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260928 12:33:30.468643 23518 raft_consensus.cc:482] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:33:30.468788 23518 raft_consensus.cc:504] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:30.469141 23518 leader_election.cc:288] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479), 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421)
Sep 28 12:33:30 dist-test-slave-g31s krb5kdc[21943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.2@KRBTEST.COM
Sep 28 12:33:30 dist-test-slave-g31s krb5kdc[21943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.3@KRBTEST.COM
I20260928 12:33:30.479513 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:30.469447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:43401 (local address 127.8.13.3:37479)
0928 12:33:30.469596 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:30.469624 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:30.469650 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:33:30.470142 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:30.470145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:30.470195 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:30.470329 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:30.471071 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:30.471954 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:30.473003 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:30.473415 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:30.473517 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:30.476138 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:30.476142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:30.476142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:30.476152 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:30.477824 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:30.478289 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:30.478292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:30.478292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:30.478388 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:30.478633 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:30.478634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:30.478634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:30.478897 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:30.478968 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:30.479256 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 12:33:30.479392 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":52,"threads_started":1}
I20260928 12:33:30.479506 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:30.469384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:37479 (local address 127.8.13.1:43401)
0928 12:33:30.469583 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:30.469594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:30.469641 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:30.470396 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:30.470399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:30.470753 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:30.470953 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:30.470959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:30.472064 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:30.472064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:30.472914 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:30.472917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:30.473503 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:30.473504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:30.473618 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:30.474254 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:30.474259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:30.476028 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:30.477903 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:30.477914 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:30.477916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:30.478224 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:30.478458 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:30.478459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:30.478459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:30.478576 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:30.478962 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:30.478969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:30.479106 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 12:33:30.479195 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:33:30.479407 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":58,"thread_start_us":66,"threads_started":1}
I20260928 12:33:30.479516 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:30.469457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:54625 (local address 127.8.13.2:37421)
0928 12:33:30.469696 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:30.469704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:30.469722 (+ 18us) server_negotiation.cc:407] Connection header received
0928 12:33:30.470165 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:30.470168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:30.470208 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:30.470329 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:30.471066 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:30.471944 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:30.472997 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:30.473417 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:30.473517 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:30.475791 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:30.475796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:30.475797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:30.475811 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:30.477789 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:30.478497 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:30.478500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:30.478500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:30.478589 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:30.478751 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:30.478752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:30.478752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:30.478987 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:30.479056 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:30.479266 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 12:33:30.479389 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
I20260928 12:33:30.479676 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:30.469358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:37421 (local address 127.8.13.1:54625)
0928 12:33:30.469516 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:30.469528 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:30.469589 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:30.470396 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:30.470399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:30.470755 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:30.470953 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:30.470959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:30.472035 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:30.472036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:30.472914 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:30.472916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:30.473500 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:30.473501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:30.473620 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:30.474185 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:30.474195 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:30.475676 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:30.478295 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:30.478295 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:30.478296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:30.478424 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:30.478648 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:30.478649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:30.478649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:30.478699 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:30.479050 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:30.479051 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:30.479118 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 12:33:30.479194 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:33:30.479486 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":180,"thread_start_us":55,"threads_started":1}
W20260928 12:33:30.480508 23133 leader_election.cc:341] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421): Not found: Tablet not found: 2df3ba1d9cca4445a46f9e5dc7f7cb53
W20260928 12:33:30.480605 23131 leader_election.cc:341] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479): Not found: Tablet not found: 2df3ba1d9cca4445a46f9e5dc7f7cb53
I20260928 12:33:30.480669 23131 leader_election.cc:302] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15322d840f904ab8b3d71afe43387c80; no voters: 629d0005d3e74139a1f06e3f829db5e2, f158271212e04ec58df6d46ef2bbbd62
I20260928 12:33:30.480793 23518 raft_consensus.cc:2726] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:33:30.735183 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:30.735014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:35321)
0928 12:33:30.735087 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:30.735137 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 12:33:30.771296 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:30.771034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:33497)
0928 12:33:30.771164 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:30.771225 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260928 12:33:30.892606 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:30.892353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:49461)
0928 12:33:30.892497 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:30.892555 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260928 12:33:31.736037 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:31.735753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:43217)
0928 12:33:31.735923 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:31.735961 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260928 12:33:31.772017 23526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:31.771793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:50613)
0928 12:33:31.771925 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:31.771969 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260928 12:33:31.893435 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:31.893171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:60559)
0928 12:33:31.893330 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:31.893385 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260928 12:33:32.141325 23528 raft_consensus.cc:482] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:33:32.141430 23528 raft_consensus.cc:504] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:32.141585 23528 leader_election.cc:288] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479), 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421)
W20260928 12:33:32.142064 23133 leader_election.cc:341] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421): Not found: Tablet not found: 2df3ba1d9cca4445a46f9e5dc7f7cb53
W20260928 12:33:32.142192 23131 leader_election.cc:341] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479): Not found: Tablet not found: 2df3ba1d9cca4445a46f9e5dc7f7cb53
I20260928 12:33:32.142243 23131 leader_election.cc:302] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15322d840f904ab8b3d71afe43387c80; no voters: 629d0005d3e74139a1f06e3f829db5e2, f158271212e04ec58df6d46ef2bbbd62
I20260928 12:33:32.142315 23528 raft_consensus.cc:2726] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:33:32.736754 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:32.736512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:33715)
0928 12:33:32.736645 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:32.736691 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260928 12:33:32.772819 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:32.772548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:51911)
0928 12:33:32.772696 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:32.772734 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260928 12:33:32.894158 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:32.893948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:60661)
0928 12:33:32.894067 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:32.894110 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260928 12:33:33.737525 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:33.737285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:53997)
0928 12:33:33.737430 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:33.737473 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260928 12:33:33.773613 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:33.773356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:55659)
0928 12:33:33.773513 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:33.773563 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260928 12:33:33.894858 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:33.894615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:46921)
0928 12:33:33.894765 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:33.894811 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260928 12:33:34.738350 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:34.738101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:33251)
0928 12:33:34.738246 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:34.738300 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 12:33:34.774365 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:34.774160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:44203)
0928 12:33:34.774273 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:34.774317 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260928 12:33:34.895529 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:34.895310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:47675)
0928 12:33:34.895436 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:34.895480 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260928 12:33:35.441991 23538 raft_consensus.cc:482] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:33:35.442090 23538 raft_consensus.cc:504] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:35.442251 23538 leader_election.cc:288] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479), 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421)
W20260928 12:33:35.442653 23133 leader_election.cc:341] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421): Not found: Tablet not found: 2df3ba1d9cca4445a46f9e5dc7f7cb53
W20260928 12:33:35.442746 23131 leader_election.cc:341] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479): Not found: Tablet not found: 2df3ba1d9cca4445a46f9e5dc7f7cb53
I20260928 12:33:35.442791 23131 leader_election.cc:302] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15322d840f904ab8b3d71afe43387c80; no voters: 629d0005d3e74139a1f06e3f829db5e2, f158271212e04ec58df6d46ef2bbbd62
I20260928 12:33:35.442881 23538 raft_consensus.cc:2726] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:33:35.739071 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:35.738848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:36411)
0928 12:33:35.738982 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:35.739021 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260928 12:33:35.775103 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:35.774852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:58715)
0928 12:33:35.775006 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:35.775052 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260928 12:33:35.896098 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:35.895901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:54853)
0928 12:33:35.896017 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:35.896051 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260928 12:33:36.739853 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:36.739578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:58171)
0928 12:33:36.739748 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:36.739799 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260928 12:33:36.775897 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:36.775657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:42581)
0928 12:33:36.775801 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:36.775848 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260928 12:33:36.896883 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:36.896620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:55919)
0928 12:33:36.896770 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:36.896817 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260928 12:33:37.740705 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:37.740481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:54033)
0928 12:33:37.740607 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:37.740655 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260928 12:33:37.776667 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:37.776438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:43501)
0928 12:33:37.776576 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:37.776618 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260928 12:33:37.897768 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:37.897495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:53399)
0928 12:33:37.897663 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:37.897714 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260928 12:33:38.741637 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:38.741384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:41295)
0928 12:33:38.741538 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:38.741585 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260928 12:33:38.777295 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:38.777084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:44297)
0928 12:33:38.777208 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:38.777240 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260928 12:33:38.898564 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:38.898319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:45781)
0928 12:33:38.898469 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:38.898514 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260928 12:33:39.595177 23551 raft_consensus.cc:482] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:33:39.595278 23551 raft_consensus.cc:504] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:39.595423 23551 leader_election.cc:288] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479), 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421)
W20260928 12:33:39.595786 23131 leader_election.cc:341] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479): Not found: Tablet not found: 2df3ba1d9cca4445a46f9e5dc7f7cb53
W20260928 12:33:39.595916 23133 leader_election.cc:341] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421): Not found: Tablet not found: 2df3ba1d9cca4445a46f9e5dc7f7cb53
I20260928 12:33:39.596038 23133 leader_election.cc:302] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15322d840f904ab8b3d71afe43387c80; no voters: 629d0005d3e74139a1f06e3f829db5e2, f158271212e04ec58df6d46ef2bbbd62
I20260928 12:33:39.596117 23551 raft_consensus.cc:2726] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:33:39.742296 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:39.742066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:59005)
0928 12:33:39.742200 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:39.742246 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260928 12:33:39.778149 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:39.777906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:37111)
0928 12:33:39.778024 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:39.778071 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260928 12:33:39.899303 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:39.899063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:48381)
0928 12:33:39.899207 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:39.899254 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260928 12:33:40.743054 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:40.742811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:49977)
0928 12:33:40.742949 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:40.742999 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260928 12:33:40.778954 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:40.778724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:53813)
0928 12:33:40.778856 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:40.778903 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260928 12:33:40.900049 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:40.899825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:37403)
0928 12:33:40.899952 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:40.899997 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260928 12:33:41.743817 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:41.743577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:45797)
0928 12:33:41.743718 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:41.743767 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260928 12:33:41.779774 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:41.779529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:50101)
0928 12:33:41.779674 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:41.779724 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260928 12:33:41.900767 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:41.900518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:46979)
0928 12:33:41.900672 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:41.900715 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260928 12:33:42.180383 23561 raft_consensus.cc:482] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:33:42.180501 23561 raft_consensus.cc:504] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f158271212e04ec58df6d46ef2bbbd62" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37479 } } peers { permanent_uuid: "629d0005d3e74139a1f06e3f829db5e2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 37421 } } peers { permanent_uuid: "15322d840f904ab8b3d71afe43387c80" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 39173 } }
I20260928 12:33:42.180692 23561 leader_election.cc:288] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479), 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421)
W20260928 12:33:42.181092 23133 leader_election.cc:341] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 629d0005d3e74139a1f06e3f829db5e2 (127.8.13.2:37421): Not found: Tablet not found: 2df3ba1d9cca4445a46f9e5dc7f7cb53
W20260928 12:33:42.181180 23131 leader_election.cc:341] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f158271212e04ec58df6d46ef2bbbd62 (127.8.13.3:37479): Not found: Tablet not found: 2df3ba1d9cca4445a46f9e5dc7f7cb53
I20260928 12:33:42.181209 23131 leader_election.cc:302] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15322d840f904ab8b3d71afe43387c80; no voters: 629d0005d3e74139a1f06e3f829db5e2, f158271212e04ec58df6d46ef2bbbd62
I20260928 12:33:42.181277 23561 raft_consensus.cc:2726] T 2df3ba1d9cca4445a46f9e5dc7f7cb53 P 15322d840f904ab8b3d71afe43387c80 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:33:42.744580 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:42.744348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:46229)
0928 12:33:42.744486 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:42.744530 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260928 12:33:42.780521 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:42.780283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:59755)
0928 12:33:42.780401 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:42.780455 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260928 12:33:42.901588 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:42.901349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:59441)
0928 12:33:42.901492 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:42.901539 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260928 12:33:43.745404 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:43.745114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.1:41347)
0928 12:33:43.745314 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:43.745356 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":58,"threads_started":1}
I20260928 12:33:43.781373 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:43.781073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.2:40141)
0928 12:33:43.781270 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:43.781323 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":65,"threads_started":1}
I20260928 12:33:43.899546 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 23087
I20260928 12:33:43.902401 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:43.902152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34429 (local address 127.8.13.3:42731)
0928 12:33:43.902318 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:43.902353 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260928 12:33:43.904251 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 23248
I20260928 12:33:43.908177 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 23382
2026-09-28T12:33:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18566 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:33:43 dist-test-slave-g31s krb5kdc[23570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:33:43 dist-test-slave-g31s krb5kdc[23570](info): set up 2 sockets
Sep 28 12:33:43 dist-test-slave-g31s krb5kdc[23570](info): commencing operation
krb5kdc: starting...
W20260928 12:33:44.853601 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:33:44 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:33:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:33:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:33:44.966933 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:37829
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:37829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:45.046289 23583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:45.046525 23583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:45.046547 23583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:45.047778 23583 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 12:33:45.047827 23583 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 12:33:45.047839 23583 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 12:33:45.047849 23583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:33:45.047863 23583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:45.047887 23583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:45.047911 23583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:33:45.047933 23583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:33:45.049530 23583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:37829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:37829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:45.049782 23583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:45.050048 23583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:45.055034 23589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:45.055034 23588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:45.065582 23591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:45.065601 23583 server_base.cc:1034] running on GCE node
I20260928 12:33:45.065986 23583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:45.066314 23583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:45.067464 23583 hybrid_clock.cc:648] HybridClock initialized: now 1790598825067450 us; error 33 us; skew 500 ppm
Sep 28 12:33:45 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:45.069977 23583 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:33:45.070484 23583 webserver.cc:460] Webserver started at http://127.8.13.62:45111/ using document root <none> and password file <none>
I20260928 12:33:45.070693 23583 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:45.070739 23583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:45.070827 23583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:45.071828 23583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "9a6041e25b004635af90249690bc5317"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "fe5a972f34cb11eb54368fc763984ce1"
server_key_iv: "957ace7f19ec3f0f1d8fc885e339a898"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.072121 23583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "9a6041e25b004635af90249690bc5317"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "fe5a972f34cb11eb54368fc763984ce1"
server_key_iv: "957ace7f19ec3f0f1d8fc885e339a898"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.073328 23583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:33:45.074040 23598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:45.074195 23583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:45.074283 23583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "9a6041e25b004635af90249690bc5317"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "fe5a972f34cb11eb54368fc763984ce1"
server_key_iv: "957ace7f19ec3f0f1d8fc885e339a898"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.074350 23583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:45.086440 23583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:45.088251 23583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:45.088369 23583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:45.092190 23583 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:37829
I20260928 12:33:45.092239 23650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:37829 every 8 connection(s)
I20260928 12:33:45.092571 23583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:33:45.093454 23651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:45.095767 23651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Bootstrap starting.
I20260928 12:33:45.096314 23651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:45.096546 23651 log.cc:824] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:45.097170 23651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: No bootstrap required, opened a new log
I20260928 12:33:45.098801 23651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:33:45.098956 23651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:45.098979 23651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a6041e25b004635af90249690bc5317, State: Initialized, Role: FOLLOWER
I20260928 12:33:45.099112 23651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:33:45.099203 23651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:33:45.099229 23651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:33:45.099260 23651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:45.099781 23651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:33:45.099874 23651 leader_election.cc:302] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a6041e25b004635af90249690bc5317; no voters:
I20260928 12:33:45.100023 23651 leader_election.cc:288] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:33:45.100077 23654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:33:45.100250 23654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 1 LEADER]: Becoming Leader. State: Replica: 9a6041e25b004635af90249690bc5317, State: Running, Role: LEADER
I20260928 12:33:45.100282 23651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:33:45.100399 23654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:33:45.100790 23655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a6041e25b004635af90249690bc5317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } } }
I20260928 12:33:45.100775 23656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a6041e25b004635af90249690bc5317. Latest consensus state: current_term: 1 leader_uuid: "9a6041e25b004635af90249690bc5317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } } }
I20260928 12:33:45.100872 23655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:45.100878 23656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: This master's current role is: LEADER
I20260928 12:33:45.101402 23660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:33:45.101966 23660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:33:45.102015 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 23583
I20260928 12:33:45.102120 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:33:45.102322 8244 external_mini_cluster.cc:1428] Setting key d470bd051ee13bc17e1ca5ed49b266cb
I20260928 12:33:45.103509 23660 catalog_manager.cc:1347] Generated new cluster ID: 2b971c0008654fd09daa16029687aff6
I20260928 12:33:45.103557 23660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:33:45 dist-test-slave-g31s krb5kdc[23570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:45.113875 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.102975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.8.13.62:37829)
0928 12:33:45.103160 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:45.103170 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:45.103211 (+ 41us) server_negotiation.cc:407] Connection header received
0928 12:33:45.103333 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:45.103340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:45.103395 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:45.103535 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:45.104290 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.105680 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.106764 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.107379 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.107475 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.110324 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:45.110330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:45.110330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:45.110344 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:45.112249 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:45.112616 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:45.112620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:45.112620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:45.112738 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:45.112901 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:45.112902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:45.112902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:45.113213 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:45.113355 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:45.113433 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:33:45.113571 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260928 12:33:45.117177 23660 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:33:45.117892 23660 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:33:45.126160 23660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Generated new TSK 0
I20260928 12:33:45.126338 23660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:33:45.158676 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:37829
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:45.238987 23677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:45.239229 23677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:45.239264 23677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:45.240513 23677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:45.240567 23677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:45.240607 23677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:33:45.242189 23677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:45.242460 23677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:45.242687 23677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:45.247718 23685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:45.247720 23683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:45.247720 23682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:45.248025 23677 server_base.cc:1034] running on GCE node
I20260928 12:33:45.248243 23677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:45.248502 23677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:45.249671 23677 hybrid_clock.cc:648] HybridClock initialized: now 1790598825249653 us; error 32 us; skew 500 ppm
Sep 28 12:33:45 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:45.252188 23677 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:33:45.252642 23677 webserver.cc:460] Webserver started at http://127.8.13.1:44457/ using document root <none> and password file <none>
I20260928 12:33:45.252853 23677 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:45.252892 23677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:45.252987 23677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:45.254043 23677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "27e95c13fe1d4128add928e345ef9565"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "9cb264776e9d19f35ad2523aec4a4029"
server_key_iv: "3ee1eeb6065fc75fe5b138975739b392"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.254340 23677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "27e95c13fe1d4128add928e345ef9565"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "9cb264776e9d19f35ad2523aec4a4029"
server_key_iv: "3ee1eeb6065fc75fe5b138975739b392"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.255503 23677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:33:45.256187 23692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:45.256340 23677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:33:45.256412 23677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "27e95c13fe1d4128add928e345ef9565"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "9cb264776e9d19f35ad2523aec4a4029"
server_key_iv: "3ee1eeb6065fc75fe5b138975739b392"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.256466 23677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:45.273216 23677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:45.275418 23677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:45.275532 23677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:45.275727 23677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:45.275981 23677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:45.276010 23677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:45.276038 23677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:45.276054 23677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:45.281486 23677 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:46363
I20260928 12:33:45.281530 23805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:46363 every 8 connection(s)
I20260928 12:33:45.281877 23677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:33:45.283883 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 23677
I20260928 12:33:45.284004 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:33:45.284219 8244 external_mini_cluster.cc:1428] Setting key b6984e5d44b733d970f87810c6606a03
Sep 28 12:33:45 dist-test-slave-g31s krb5kdc[23570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:45.292454 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.282638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:59667 (local address 127.8.13.62:37829)
0928 12:33:45.282707 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:45.282709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:45.282896 (+ 187us) server_negotiation.cc:407] Connection header received
0928 12:33:45.283420 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:45.283422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:45.283432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:45.283484 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:45.284708 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.285290 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.286659 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.286975 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.287051 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.289429 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:45.289432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:45.289432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:45.289447 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:45.291033 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:45.291478 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:45.291480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:45.291481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:45.291541 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:45.291773 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:45.291774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:45.291774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:45.291930 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:45.292002 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:45.292256 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 12:33:45.292354 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:33:45.292531 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.282596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:59667)
0928 12:33:45.282779 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:45.282817 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:45.283327 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:45.283557 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:45.283561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:45.284314 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:45.284605 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.284613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.285389 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.285390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.286579 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.286582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.287050 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.287051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.287148 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.287701 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:45.287710 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:45.289317 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:45.291139 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:45.291148 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:45.291150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:45.291404 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:45.291605 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:45.291606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:45.291606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:45.291715 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:45.291989 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:45.291992 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:45.292123 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 12:33:45.292202 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 12:33:45.292339 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
I20260928 12:33:45.293071 23806 heartbeater.cc:347] Connected to a master server at 127.8.13.62:37829
I20260928 12:33:45.293180 23806 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:45.293377 23806 heartbeater.cc:511] Master 127.8.13.62:37829 requested a full tablet report, sending...
I20260928 12:33:45.293926 23615 ts_manager.cc:194] Registered new tserver with Master: 27e95c13fe1d4128add928e345ef9565 (127.8.13.1:46363)
I20260928 12:33:45.294546 23615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:59667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:33:45.310289 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:37829
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:45.390408 23811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:45.390632 23811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:45.390666 23811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:45.391909 23811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:45.391968 23811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:45.392005 23811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:33:45.393566 23811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:45.393862 23811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:45.394093 23811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:45.399075 23816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:45.399066 23819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:45.399047 23817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:45.399207 23811 server_base.cc:1034] running on GCE node
I20260928 12:33:45.399537 23811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:45.399777 23811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:45.400923 23811 hybrid_clock.cc:648] HybridClock initialized: now 1790598825400909 us; error 30 us; skew 500 ppm
Sep 28 12:33:45 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:45.403321 23811 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:33:45.403719 23811 webserver.cc:460] Webserver started at http://127.8.13.2:33289/ using document root <none> and password file <none>
I20260928 12:33:45.403898 23811 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:45.403937 23811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:45.404024 23811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:45.405052 23811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "158ce3394b99467ca77a863028408fb5"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "eadeb82be1e22382aa49e0c78844534d"
server_key_iv: "4aea462aaec4d0e3a556fd4f37f6cebb"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.405333 23811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "158ce3394b99467ca77a863028408fb5"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "eadeb82be1e22382aa49e0c78844534d"
server_key_iv: "4aea462aaec4d0e3a556fd4f37f6cebb"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.406572 23811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:33:45.407176 23826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:45.407352 23811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:33:45.407428 23811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "158ce3394b99467ca77a863028408fb5"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "eadeb82be1e22382aa49e0c78844534d"
server_key_iv: "4aea462aaec4d0e3a556fd4f37f6cebb"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.407482 23811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:45.417712 23811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:45.419709 23811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:45.419818 23811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:45.419992 23811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:45.420224 23811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:45.420251 23811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:45.420272 23811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:45.420287 23811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:45.425464 23811 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:42285
I20260928 12:33:45.425527 23939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:42285 every 8 connection(s)
I20260928 12:33:45.425792 23811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:33:45 dist-test-slave-g31s krb5kdc[23570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:45.433786 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.426507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:51083 (local address 127.8.13.62:37829)
0928 12:33:45.426555 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:45.426557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:45.426761 (+ 204us) server_negotiation.cc:407] Connection header received
0928 12:33:45.427215 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:45.427216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:45.427226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:45.427276 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:45.428131 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.428643 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.429618 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.429809 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.429852 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.431698 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:45.431699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:45.431699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:45.431705 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:45.432738 (+ 1033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:45.433075 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:45.433075 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:45.433076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:45.433121 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:45.433278 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:45.433278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:45.433278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:45.433372 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:45.433414 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:45.433647 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 12:33:45.433718 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:33:45.433981 23942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.426486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:51083)
0928 12:33:45.426651 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:45.426685 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:45.427141 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:45.427333 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:45.427336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:45.427876 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:45.428061 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.428066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.428743 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.428744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.429551 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.429554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.429862 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.429863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.429963 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.430363 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:45.430371 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:45.431619 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:45.432806 (+ 1187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:45.432812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:45.432814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:45.433019 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:45.433158 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:45.433158 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:45.433159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:45.433239 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:45.433429 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:45.433433 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:45.433550 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 12:33:45.433627 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 12:33:45.433744 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260928 12:33:45.434437 23940 heartbeater.cc:347] Connected to a master server at 127.8.13.62:37829
I20260928 12:33:45.434528 23940 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:45.434715 23940 heartbeater.cc:511] Master 127.8.13.62:37829 requested a full tablet report, sending...
I20260928 12:33:45.434762 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 23811
I20260928 12:33:45.434875 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:33:45.435081 8244 external_mini_cluster.cc:1428] Setting key c0f49201cbc809a88063caeda26e7967
I20260928 12:33:45.435091 23615 ts_manager.cc:194] Registered new tserver with Master: 158ce3394b99467ca77a863028408fb5 (127.8.13.2:42285)
I20260928 12:33:45.435573 23615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:51083
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:33:45.460412 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:37829
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:33:45.540303 23945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:33:45.540529 23945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:33:45.540555 23945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:33:45.541776 23945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:33:45.541853 23945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:33:45.541896 23945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:33:45.543457 23945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:33:45.543720 23945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:33:45.543949 23945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:33:45.549063 23953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:33:45.549063 23951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:45.549124 23945 server_base.cc:1034] running on GCE node
W20260928 12:33:45.549094 23950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:33:45.549505 23945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:33:45.549767 23945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:33:45.550966 23945 hybrid_clock.cc:648] HybridClock initialized: now 1790598825550948 us; error 35 us; skew 500 ppm
Sep 28 12:33:45 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:33:45.553565 23945 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:33:45.554078 23945 webserver.cc:460] Webserver started at http://127.8.13.3:40865/ using document root <none> and password file <none>
I20260928 12:33:45.554312 23945 fs_manager.cc:359] Metadata directory not provided
I20260928 12:33:45.554364 23945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:33:45.554498 23945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:33:45.555611 23945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "70f6e5cb62ec47f1846df79d0a91c847"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "e4276a5bdea25ea14a92b22c5aad15ce"
server_key_iv: "d6307a630a01600b102c7a689841b76f"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.555979 23945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "70f6e5cb62ec47f1846df79d0a91c847"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "e4276a5bdea25ea14a92b22c5aad15ce"
server_key_iv: "d6307a630a01600b102c7a689841b76f"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.557391 23945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:33:45.558326 23960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:45.558508 23945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:33:45.558611 23945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "70f6e5cb62ec47f1846df79d0a91c847"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "e4276a5bdea25ea14a92b22c5aad15ce"
server_key_iv: "d6307a630a01600b102c7a689841b76f"
server_key_version: "encryptionkey@0"
I20260928 12:33:45.558686 23945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:33:45.569988 23945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:33:45.572247 23945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:33:45.572407 23945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:33:45.572654 23945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:33:45.572968 23945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:33:45.573004 23945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:45.573035 23945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:33:45.573060 23945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:33:45.579100 23945 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:33681
I20260928 12:33:45.579139 24073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:33681 every 8 connection(s)
I20260928 12:33:45.579483 23945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:33:45.585322 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 23945
I20260928 12:33:45.585448 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:33:45.585657 8244 external_mini_cluster.cc:1428] Setting key ce0d4071f488748b60b8980670873fe4
Sep 28 12:33:45 dist-test-slave-g31s krb5kdc[23570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:33:45.588869 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.580205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:48265 (local address 127.8.13.62:37829)
0928 12:33:45.580254 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:45.580256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:45.580481 (+ 225us) server_negotiation.cc:407] Connection header received
0928 12:33:45.581411 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:45.581412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:45.581424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:45.581484 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:45.582215 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.582738 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.583785 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.583972 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.584027 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.586457 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:45.586459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:45.586459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:45.586468 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:45.587543 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:45.587928 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:45.587929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:45.587930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:45.587984 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:45.588196 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:45.588197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:45.588197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:45.588356 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:45.588413 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:45.588689 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 12:33:45.588773 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:33:45.589088 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.580175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:48265)
0928 12:33:45.580339 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:45.580382 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:45.581307 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:45.581542 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:45.581545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:45.581872 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:33:45.582088 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.582093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.582838 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.582839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.583697 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.583699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.584053 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.584054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.584161 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.584700 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:33:45.584708 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:33:45.586366 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:33:45.587619 (+ 1253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:45.587627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:45.587630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:45.587859 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:45.588037 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:33:45.588037 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:33:45.588038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:33:45.588143 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:33:45.588417 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:33:45.588423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:33:45.588569 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 12:33:45.588661 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 12:33:45.588856 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260928 12:33:45.589586 24074 heartbeater.cc:347] Connected to a master server at 127.8.13.62:37829
I20260928 12:33:45.589674 24074 heartbeater.cc:464] Registering TS with master...
I20260928 12:33:45.589890 24074 heartbeater.cc:511] Master 127.8.13.62:37829 requested a full tablet report, sending...
I20260928 12:33:45.590257 23615 ts_manager.cc:194] Registered new tserver with Master: 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681)
I20260928 12:33:45.590621 23615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:48265
I20260928 12:33:45.597553 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:33:45.604808 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.598920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.8.13.62:37829)
0928 12:33:45.598971 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:45.598973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:45.599108 (+ 135us) server_negotiation.cc:407] Connection header received
0928 12:33:45.599137 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:45.599138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:45.599150 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:45.599234 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:33:45.599827 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.600416 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.601148 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.601362 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.601412 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.602667 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:33:45.602668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:33:45.602669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:33:45.602676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:33:45.603798 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:45.604086 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:45.604087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:45.604087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:45.604140 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:33:45.604301 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:33:45.604302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:33:45.604302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:33:45.604449 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:33:45.604532 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:45.604639 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 12:33:45.604713 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:33:45.606241 23615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:33:45.607802 23615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:33:45.613883 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.609819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:33681 (local address 127.0.0.1:58518)
0928 12:33:45.610153 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:45.610170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:45.610232 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:45.610500 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:45.610503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:45.610505 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:45.610583 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.610585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.611901 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.611902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.612956 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.612957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.613563 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.613564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.613649 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.613661 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 12:33:45.613724 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 12:33:45.613774 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":156,"thread_start_us":56,"threads_started":1}
I20260928 12:33:45.614004 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.609922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.8.13.1:46363)
0928 12:33:45.610131 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:45.610136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:45.610306 (+ 170us) server_negotiation.cc:407] Connection header received
0928 12:33:45.610346 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:45.610348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:45.610361 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:45.610436 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:45.610878 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.611913 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.612928 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.613618 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.613707 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.613769 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:45.613866 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 12:33:45.613909 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260928 12:33:45.614320 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.609935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.8.13.3:33681)
0928 12:33:45.610308 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:45.610311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:45.610319 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:33:45.610352 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:45.610354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:45.610368 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:45.610430 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:45.610652 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.611806 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.613035 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.613489 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.613549 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.613608 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:45.614226 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 12:33:45.614261 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":92,"threads_started":1}
I20260928 12:33:45.614367 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.609954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.8.13.2:42285)
0928 12:33:45.610210 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:45.610215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:45.610226 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:33:45.610342 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:45.610346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:45.610361 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:45.610433 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:45.610778 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.611550 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.612985 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.613644 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.613904 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.613962 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:45.614276 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 12:33:45.614312 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":75,"threads_started":1}
I20260928 12:33:45.613965 24082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.609792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:46363 (local address 127.0.0.1:45606)
0928 12:33:45.610098 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:45.610166 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:45.610232 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:45.610776 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:45.610776 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:45.610778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:45.610803 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.610804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.612003 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.612004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.612858 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.612860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.613709 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.613709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.613762 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.613765 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:33:45.613850 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 12:33:45.613882 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":5,"thread_start_us":54,"threads_started":1}
I20260928 12:33:45.614225 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.609819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:42285 (local address 127.0.0.1:59002)
0928 12:33:45.610170 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:45.610178 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:45.610232 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:45.610654 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:45.610655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:45.610657 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:45.610706 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.610708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.611634 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.611635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.612913 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.612916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.613996 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.613998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.614059 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.614063 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:33:45.614111 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:33:45.614150 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":280,"thread_start_us":37,"threads_started":1}
I20260928 12:33:45.615108 24008 tablet_service.cc:1467] Processing CreateTablet for tablet d0ceb777aa3c4b5897d20684b1fe262b (DEFAULT_TABLE table=test-table [id=99e2fd6390964c248bbc9daa767b3200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:45.615173 23740 tablet_service.cc:1467] Processing CreateTablet for tablet d0ceb777aa3c4b5897d20684b1fe262b (DEFAULT_TABLE table=test-table [id=99e2fd6390964c248bbc9daa767b3200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:45.615177 23874 tablet_service.cc:1467] Processing CreateTablet for tablet d0ceb777aa3c4b5897d20684b1fe262b (DEFAULT_TABLE table=test-table [id=99e2fd6390964c248bbc9daa767b3200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:33:45.615425 24008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ceb777aa3c4b5897d20684b1fe262b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:45.615484 23740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ceb777aa3c4b5897d20684b1fe262b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:45.615484 23874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ceb777aa3c4b5897d20684b1fe262b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:33:45.617884 24088 tablet_bootstrap.cc:492] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847: Bootstrap starting.
I20260928 12:33:45.618520 24089 tablet_bootstrap.cc:492] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: Bootstrap starting.
I20260928 12:33:45.618563 24088 tablet_bootstrap.cc:654] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:45.618753 24090 tablet_bootstrap.cc:492] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Bootstrap starting.
I20260928 12:33:45.619139 24088 log.cc:824] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:45.619230 24089 tablet_bootstrap.cc:654] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:45.619266 24090 tablet_bootstrap.cc:654] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Neither blocks nor log segments found. Creating new log.
I20260928 12:33:45.619704 24089 log.cc:824] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:45.619735 24090 log.cc:824] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Log is configured to *not* fsync() on all Append() calls
I20260928 12:33:45.620325 24089 tablet_bootstrap.cc:492] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: No bootstrap required, opened a new log
I20260928 12:33:45.620344 24088 tablet_bootstrap.cc:492] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847: No bootstrap required, opened a new log
I20260928 12:33:45.620386 24088 ts_tablet_manager.cc:1397] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 12:33:45.620384 24089 ts_tablet_manager.cc:1397] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:33:45.621910 24088 raft_consensus.cc:348] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } }
I20260928 12:33:45.621891 24089 raft_consensus.cc:348] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } }
I20260928 12:33:45.622051 24088 raft_consensus.cc:374] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:45.622052 24089 raft_consensus.cc:374] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:45.622085 24088 raft_consensus.cc:729] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f6e5cb62ec47f1846df79d0a91c847, State: Initialized, Role: FOLLOWER
I20260928 12:33:45.622085 24089 raft_consensus.cc:729] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158ce3394b99467ca77a863028408fb5, State: Initialized, Role: FOLLOWER
I20260928 12:33:45.622162 24090 tablet_bootstrap.cc:492] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: No bootstrap required, opened a new log
I20260928 12:33:45.622203 24089 consensus_queue.cc:260] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } }
I20260928 12:33:45.622270 24090 ts_tablet_manager.cc:1397] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 12:33:45.622254 24088 consensus_queue.cc:260] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } }
I20260928 12:33:45.622479 24089 ts_tablet_manager.cc:1428] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:33:45.622489 24088 ts_tablet_manager.cc:1428] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:33:45.622519 23940 heartbeater.cc:503] Master 127.8.13.62:37829 was elected leader, sending a full tablet report...
I20260928 12:33:45.622763 24074 heartbeater.cc:503] Master 127.8.13.62:37829 was elected leader, sending a full tablet report...
I20260928 12:33:45.624055 24090 raft_consensus.cc:348] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } }
I20260928 12:33:45.624191 24090 raft_consensus.cc:374] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:33:45.624212 24090 raft_consensus.cc:729] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e95c13fe1d4128add928e345ef9565, State: Initialized, Role: FOLLOWER
I20260928 12:33:45.624293 24090 consensus_queue.cc:260] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } }
I20260928 12:33:45.624519 24090 ts_tablet_manager.cc:1428] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:33:45.624571 23806 heartbeater.cc:503] Master 127.8.13.62:37829 was elected leader, sending a full tablet report...
W20260928 12:33:45.676397 23941 tablet.cc:2369] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:33:45.689000 24094 raft_consensus.cc:482] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:33:45.689136 24094 raft_consensus.cc:504] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } }
I20260928 12:33:45.689565 24094 leader_election.cc:288] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27e95c13fe1d4128add928e345ef9565 (127.8.13.1:46363), 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681)
I20260928 12:33:45.693559 23942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.689778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:33681 (local address 127.8.13.2:44529)
0928 12:33:45.689934 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:45.689945 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:45.690006 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:45.690168 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:45.690169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:45.690171 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:45.690257 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.690259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.691152 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.691153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.692462 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.692466 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.693263 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.693264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.693350 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.693353 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:33:45.693407 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:33:45.693456 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":41}
I20260928 12:33:45.693585 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.689913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:44529 (local address 127.8.13.3:33681)
0928 12:33:45.689986 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:45.689988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:45.689998 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:33:45.690078 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:45.690079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:45.690093 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:45.690157 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:45.690323 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.691073 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.692582 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.693175 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.693257 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.693268 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:45.693453 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 12:33:45.693509 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:33:45.693589 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.689850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:42419 (local address 127.8.13.1:46363)
0928 12:33:45.689903 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:45.689905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:45.690073 (+ 168us) server_negotiation.cc:407] Connection header received
0928 12:33:45.690106 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:45.690107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:45.690119 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:45.690170 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:33:45.690324 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.691068 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.692582 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.693176 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.693257 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.693268 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:45.693453 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 12:33:45.693509 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:33:45.693564 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.689778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:46363 (local address 127.8.13.2:42419)
0928 12:33:45.690000 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:45.690008 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:33:45.690058 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:33:45.690179 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:33:45.690179 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:33:45.690180 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:33:45.690261 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.690262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.691153 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.691153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.692462 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:33:45.692466 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.693263 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.693264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:33:45.693350 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.693354 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:33:45.693407 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:33:45.693456 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":134,"thread_start_us":113,"threads_started":1}
I20260928 12:33:45.693953 23760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ceb777aa3c4b5897d20684b1fe262b" candidate_uuid: "158ce3394b99467ca77a863028408fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e95c13fe1d4128add928e345ef9565" is_pre_election: true
I20260928 12:33:45.693953 24028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ceb777aa3c4b5897d20684b1fe262b" candidate_uuid: "158ce3394b99467ca77a863028408fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f6e5cb62ec47f1846df79d0a91c847" is_pre_election: true
I20260928 12:33:45.694121 23760 raft_consensus.cc:2454] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158ce3394b99467ca77a863028408fb5 in term 0.
I20260928 12:33:45.694121 24028 raft_consensus.cc:2454] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158ce3394b99467ca77a863028408fb5 in term 0.
I20260928 12:33:45.694403 23827 leader_election.cc:302] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158ce3394b99467ca77a863028408fb5, 70f6e5cb62ec47f1846df79d0a91c847; no voters:
I20260928 12:33:45.694582 24094 raft_consensus.cc:2781] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:33:45.694638 24094 raft_consensus.cc:482] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:33:45.694653 24094 raft_consensus.cc:3037] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:45.695317 24094 raft_consensus.cc:504] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } }
I20260928 12:33:45.695446 24094 leader_election.cc:288] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 1 election: Requested vote from peers 27e95c13fe1d4128add928e345ef9565 (127.8.13.1:46363), 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681)
I20260928 12:33:45.695626 23760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ceb777aa3c4b5897d20684b1fe262b" candidate_uuid: "158ce3394b99467ca77a863028408fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e95c13fe1d4128add928e345ef9565"
I20260928 12:33:45.695626 24028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ceb777aa3c4b5897d20684b1fe262b" candidate_uuid: "158ce3394b99467ca77a863028408fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f6e5cb62ec47f1846df79d0a91c847"
I20260928 12:33:45.695708 24028 raft_consensus.cc:3037] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:45.695713 23760 raft_consensus.cc:3037] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:33:45.696446 23760 raft_consensus.cc:2454] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158ce3394b99467ca77a863028408fb5 in term 1.
I20260928 12:33:45.696449 24028 raft_consensus.cc:2454] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158ce3394b99467ca77a863028408fb5 in term 1.
I20260928 12:33:45.696616 23827 leader_election.cc:302] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158ce3394b99467ca77a863028408fb5, 70f6e5cb62ec47f1846df79d0a91c847; no voters:
I20260928 12:33:45.696697 24094 raft_consensus.cc:2781] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:33:45.696847 24094 raft_consensus.cc:686] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 1 LEADER]: Becoming Leader. State: Replica: 158ce3394b99467ca77a863028408fb5, State: Running, Role: LEADER
I20260928 12:33:45.696941 24094 consensus_queue.cc:237] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } }
I20260928 12:33:45.697839 23615 catalog_manager.cc:5495] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 158ce3394b99467ca77a863028408fb5 (127.8.13.2). New cstate: current_term: 1 leader_uuid: "158ce3394b99467ca77a863028408fb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } health_report { overall_health: UNKNOWN } } }
I20260928 12:33:45.705153 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.702946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59012 (local address 127.8.13.2:42285)
0928 12:33:45.702985 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:33:45.702987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:33:45.702994 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:33:45.703058 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:33:45.703059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:33:45.703080 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:33:45.703138 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:33:45.703259 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.703739 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.704465 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:33:45.704664 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:33:45.704717 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:33:45.704862 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:33:45.704964 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:33:45.705034 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:33:45.705053 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 12:33:45.705073 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:33:45.706836 24028 raft_consensus.cc:1261] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847 [term 1 FOLLOWER]: Refusing update from remote peer 158ce3394b99467ca77a863028408fb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:33:45.706846 23760 raft_consensus.cc:1261] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Refusing update from remote peer 158ce3394b99467ca77a863028408fb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:33:45.707178 24098 consensus_queue.cc:1035] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:33:45.707319 24094 consensus_queue.cc:1035] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:33:45.709208 24101 mvcc.cc:205] Tried to move back new op lower bound from 7334292790092685312 to 7334292790055251968. Current Snapshot: MvccSnapshot[applied={T|T < 7334292790092685312}]
I20260928 12:33:45.709506 24102 mvcc.cc:205] Tried to move back new op lower bound from 7334292790092685312 to 7334292790055251968. Current Snapshot: MvccSnapshot[applied={T|T < 7334292790092685312}]
I20260928 12:33:45.709698 24103 mvcc.cc:205] Tried to move back new op lower bound from 7334292790092685312 to 7334292790055251968. Current Snapshot: MvccSnapshot[applied={T|T < 7334292790092685312}]
I20260928 12:33:45.712155 23614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:33:45.712241 23614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:33:45.712996 23614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Sending DeleteTablet for 3 replicas of tablet d0ceb777aa3c4b5897d20684b1fe262b
I20260928 12:33:45.713332 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 23677
I20260928 12:33:45.713349 23874 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ceb777aa3c4b5897d20684b1fe262b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:45 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:59002
I20260928 12:33:45.713349 24008 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ceb777aa3c4b5897d20684b1fe262b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:45 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:58518
I20260928 12:33:45.717945 24110 tablet_replica.cc:330] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: stopping tablet replica
I20260928 12:33:45.718084 24110 raft_consensus.cc:2229] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:33:45.718308 24110 raft_consensus.cc:2258] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:33:45.718551 24111 tablet_replica.cc:330] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847: stopping tablet replica
I20260928 12:33:45.718693 24111 raft_consensus.cc:2229] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:33:45.718822 24111 raft_consensus.cc:2258] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:33:45.718950 24110 ts_tablet_manager.cc:1905] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:33:45.719317 24111 ts_tablet_manager.cc:1905] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 12:33:45.719491 23600 connection.cc:535] client connection to 127.8.13.1:46363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 12:33:45.719581 23600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:33:45.719825 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 23811
I20260928 12:33:45.720881 24111 ts_tablet_manager.cc:1918] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:33:45.720944 24111 log.cc:1196] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/d0ceb777aa3c4b5897d20684b1fe262b
I20260928 12:33:45.721230 24111 ts_tablet_manager.cc:1939] T d0ceb777aa3c4b5897d20684b1fe262b P 70f6e5cb62ec47f1846df79d0a91c847: Deleting consensus metadata
I20260928 12:33:45.721596 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.721443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:46363 (local address 127.0.0.1:45616)
0928 12:33:45.721508 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:45.721557 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 12:33:45.722098 23599 catalog_manager.cc:4841] TS 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681): tablet d0ceb777aa3c4b5897d20684b1fe262b (table test-table [id=99e2fd6390964c248bbc9daa767b3200]) successfully deleted
W20260928 12:33:45.724578 23600 catalog_manager.cc:4584] TS 27e95c13fe1d4128add928e345ef9565 (127.8.13.1:46363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0ceb777aa3c4b5897d20684b1fe262b: Network error: Client connection negotiation failed: client connection to 127.8.13.1:46363: connect: Connection refused (error 111)
I20260928 12:33:45.724892 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 23945
I20260928 12:33:45.725497 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:33:45.725019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:42285 (local address 127.0.0.1:59020)
0928 12:33:45.725425 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 12:33:45.725459 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.2:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
I20260928 12:33:45.729260 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 23583
I20260928 12:34:05.734391 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:37829
--webserver_interface=127.8.13.62
--webserver_port=45111
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:37829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:05.817229 24113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:05.817430 24113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:05.817451 24113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:05.818567 24113 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 12:34:05.818600 24113 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 12:34:05.818612 24113 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 12:34:05.818622 24113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:05.818634 24113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:05.818645 24113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:05.818656 24113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:05.818673 24113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:05.820127 24113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:37829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:37829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:05.820319 24113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:05.820492 24113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:05.828364 24122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:05.828387 24119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:05.828429 24120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:05.828418 24113 server_base.cc:1034] running on GCE node
I20260928 12:34:05.828838 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:05.829174 24113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:05.830315 24113 hybrid_clock.cc:648] HybridClock initialized: now 1790598845830294 us; error 31 us; skew 500 ppm
Sep 28 12:34:05 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598845, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:05.832695 24113 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:34:05.833171 24113 webserver.cc:460] Webserver started at http://127.8.13.62:45111/ using document root <none> and password file <none>
I20260928 12:34:05.833389 24113 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:05.833453 24113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:05.834833 24113 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:05.835441 24129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:05.835588 24113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:05.835672 24113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "9a6041e25b004635af90249690bc5317"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "fe5a972f34cb11eb54368fc763984ce1"
server_key_iv: "957ace7f19ec3f0f1d8fc885e339a898"
server_key_version: "encryptionkey@0"
I20260928 12:34:05.836091 24113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:05.855295 24113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:05.857334 24113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:05.857489 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:05.861426 24113 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:37829
I20260928 12:34:05.861462 24181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:37829 every 8 connection(s)
I20260928 12:34:05.861887 24113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:05.863401 24182 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:34:05.864027 24182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Bootstrap starting.
I20260928 12:34:05.869771 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 24113
I20260928 12:34:05.870095 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:46363
--local_ip_for_outbound_sockets=127.8.13.1
--tserver_master_addrs=127.8.13.62:37829
--webserver_port=44457
--webserver_interface=127.8.13.1
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:34:05.872272 24182 log.cc:824] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:05.874377 24182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:34:05.874634 24182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Bootstrap complete.
I20260928 12:34:05.876823 24182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:34:05.877067 24182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a6041e25b004635af90249690bc5317, State: Initialized, Role: FOLLOWER
I20260928 12:34:05.877230 24182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:34:05.877311 24182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:05.877367 24182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:05.877413 24182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:34:05.878528 24182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:34:05.878631 24182 leader_election.cc:302] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a6041e25b004635af90249690bc5317; no voters:
I20260928 12:34:05.878796 24182 leader_election.cc:288] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:34:05.878854 24186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:34:05.879006 24186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 2 LEADER]: Becoming Leader. State: Replica: 9a6041e25b004635af90249690bc5317, State: Running, Role: LEADER
I20260928 12:34:05.879084 24182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:05.879238 24186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:34:05.879554 24187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9a6041e25b004635af90249690bc5317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } } }
I20260928 12:34:05.879640 24187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:05.879926 24195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:05.880033 24188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a6041e25b004635af90249690bc5317. Latest consensus state: current_term: 2 leader_uuid: "9a6041e25b004635af90249690bc5317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } } }
I20260928 12:34:05.880154 24188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:05.880792 24195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:05.880895 24195 catalog_manager.cc:1259] Loaded cluster ID: 2b971c0008654fd09daa16029687aff6
I20260928 12:34:05.880972 24195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:05.882169 24195 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:05.882366 24195 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Loaded TSK: 0
I20260928 12:34:05.897264 24195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Generated new TSK 1
I20260928 12:34:05.897395 24195 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:34:05.952033 24184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:05.952272 24184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:05.952309 24184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:05.953542 24184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:05.953594 24184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:05.953634 24184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:34:05.955295 24184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:46363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=44457
--tserver_master_addrs=127.8.13.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:05.955549 24184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:05.955770 24184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:34:05.960773 24184 server_base.cc:1034] running on GCE node
W20260928 12:34:05.960773 24209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:05.960793 24212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:05.961138 24210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:05.961380 24184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:05.961633 24184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:05.962783 24184 hybrid_clock.cc:648] HybridClock initialized: now 1790598845962768 us; error 34 us; skew 500 ppm
Sep 28 12:34:05 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598845, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:05.965472 24184 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:34:05.966001 24184 webserver.cc:460] Webserver started at http://127.8.13.1:44457/ using document root <none> and password file <none>
I20260928 12:34:05.966235 24184 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:05.966292 24184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:05.967713 24184 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:34:05.968475 24219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:05.968628 24184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:34:05.968714 24184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "27e95c13fe1d4128add928e345ef9565"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "9cb264776e9d19f35ad2523aec4a4029"
server_key_iv: "3ee1eeb6065fc75fe5b138975739b392"
server_key_version: "encryptionkey@0"
I20260928 12:34:05.969080 24184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:05.989447 24184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:05.991645 24184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:05.991765 24184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:05.992005 24184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:05.992450 24226 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:34:05.993928 24184 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:34:05.993978 24184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:34:05.994002 24184 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:34:05.994655 24184 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:34:05.994699 24184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:34:05.994828 24226 tablet_bootstrap.cc:492] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Bootstrap starting.
I20260928 12:34:06.001987 24184 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:46363
I20260928 12:34:06.002043 24333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:46363 every 8 connection(s)
I20260928 12:34:06.002707 24184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:34:06.005034 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 24184
I20260928 12:34:06.005354 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:42285
--local_ip_for_outbound_sockets=127.8.13.2
--tserver_master_addrs=127.8.13.62:37829
--webserver_port=33289
--webserver_interface=127.8.13.2
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:34:06.005774 24226 log.cc:824] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:06.008405 24226 tablet_bootstrap.cc:492] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:34:06.008754 24226 tablet_bootstrap.cc:492] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Bootstrap complete.
I20260928 12:34:06.009843 24226 ts_tablet_manager.cc:1397] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
Sep 28 12:34:06 dist-test-slave-g31s krb5kdc[23570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598845, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:06.011250 24226 raft_consensus.cc:348] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } }
I20260928 12:34:06.011557 24226 raft_consensus.cc:729] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e95c13fe1d4128add928e345ef9565, State: Initialized, Role: FOLLOWER
I20260928 12:34:06.011684 24226 consensus_queue.cc:260] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } } peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } }
I20260928 12:34:06.011955 24226 ts_tablet_manager.cc:1428] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 12:34:06.016136 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.003433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:48735)
0928 12:34:06.003701 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:06.003742 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:06.004333 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:06.005110 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:06.005114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:06.005565 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:06.005928 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.005937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.007488 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.007489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.008693 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.008697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.009576 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.009577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.009716 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.010311 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:06.010320 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:06.012180 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:06.014374 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:06.014380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:06.014382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:06.014650 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:06.014913 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:06.014913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:06.014914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:06.015030 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:06.015508 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:06.015514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:06.015670 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 12:34:06.015781 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 12:34:06.015946 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
I20260928 12:34:06.016198 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.003524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:48735 (local address 127.8.13.62:37829)
0928 12:34:06.004590 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.004601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.004656 (+ 55us) server_negotiation.cc:407] Connection header received
0928 12:34:06.004789 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.004796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.004849 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.004971 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:06.006075 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.007355 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.008787 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.009379 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.009471 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.012299 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:06.012303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:06.012304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:06.012316 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:06.014291 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.014730 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.014734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.014734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.014847 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.015097 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.015098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.015098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.015432 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:06.015519 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:06.015848 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 12:34:06.015990 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":106,"threads_started":1}
I20260928 12:34:06.016776 24334 heartbeater.cc:347] Connected to a master server at 127.8.13.62:37829
I20260928 12:34:06.016875 24334 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:06.017055 24334 heartbeater.cc:511] Master 127.8.13.62:37829 requested a full tablet report, sending...
I20260928 12:34:06.017755 24146 ts_manager.cc:194] Registered new tserver with Master: 27e95c13fe1d4128add928e345ef9565 (127.8.13.1:46363)
I20260928 12:34:06.018738 24146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:48735
I20260928 12:34:06.019670 24334 heartbeater.cc:503] Master 127.8.13.62:37829 was elected leader, sending a full tablet report...
Sep 28 12:34:06 dist-test-slave-g31s krb5kdc[23570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598845, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:34:06.026122 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.018977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.8.13.1:46363)
0928 12:34:06.019112 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.019116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.019125 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:34:06.019160 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.019163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.019176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.019239 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:06.019611 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.020472 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.021173 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.021455 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.021509 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.023552 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:06.023559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:06.023559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:06.023565 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:06.024948 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.025389 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.025391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.025391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.025442 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.025611 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.025612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.025612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.025734 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:06.025789 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:06.025982 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 12:34:06.026056 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260928 12:34:06.026173 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.018315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:46363 (local address 127.0.0.1:39598)
0928 12:34:06.018483 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:06.018517 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:06.018577 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:06.019243 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:06.019249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:06.019493 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:06.019534 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.019536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.020545 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.020546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.021094 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.021096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.021514 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.021516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.021590 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.022049 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:06.022053 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:06.023476 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:06.025025 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:06.025031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:06.025033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:06.025320 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:06.025494 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:06.025494 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:06.025495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:06.025562 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:06.025783 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:06.025788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:06.025863 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 12:34:06.025940 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 12:34:06.026062 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260928 12:34:06.026608 24265 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ceb777aa3c4b5897d20684b1fe262b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:33:45 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:39598
I20260928 12:34:06.027186 24343 tablet_replica.cc:330] stopping tablet replica
I20260928 12:34:06.027274 24343 raft_consensus.cc:2229] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:34:06.027339 24343 raft_consensus.cc:2258] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:06.027602 24343 ts_tablet_manager.cc:1905] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:06.029543 24343 ts_tablet_manager.cc:1918] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:06.029598 24343 log.cc:1196] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal/wals/d0ceb777aa3c4b5897d20684b1fe262b
I20260928 12:34:06.029814 24343 ts_tablet_manager.cc:1939] T d0ceb777aa3c4b5897d20684b1fe262b P 27e95c13fe1d4128add928e345ef9565: Deleting consensus metadata
I20260928 12:34:06.030241 24131 catalog_manager.cc:4841] TS 27e95c13fe1d4128add928e345ef9565 (127.8.13.1:46363): tablet d0ceb777aa3c4b5897d20684b1fe262b (table test-table [id=99e2fd6390964c248bbc9daa767b3200]) successfully deleted
W20260928 12:34:06.092048 24339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:06.092259 24339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:06.092291 24339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:06.093426 24339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:06.093466 24339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:06.093502 24339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:34:06.094990 24339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:42285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=33289
--tserver_master_addrs=127.8.13.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:06.095233 24339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:06.095445 24339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:06.100270 24348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:06.100260 24349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:06.100260 24351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:06.100530 24339 server_base.cc:1034] running on GCE node
I20260928 12:34:06.100734 24339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:06.100961 24339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:06.102142 24339 hybrid_clock.cc:648] HybridClock initialized: now 1790598846102129 us; error 28 us; skew 500 ppm
Sep 28 12:34:06 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598846, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:06.104532 24339 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:34:06.104974 24339 webserver.cc:460] Webserver started at http://127.8.13.2:33289/ using document root <none> and password file <none>
I20260928 12:34:06.105165 24339 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:06.105218 24339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:06.106539 24339 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:34:06.107179 24358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:06.107333 24339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:34:06.107412 24339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "158ce3394b99467ca77a863028408fb5"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "eadeb82be1e22382aa49e0c78844534d"
server_key_iv: "4aea462aaec4d0e3a556fd4f37f6cebb"
server_key_version: "encryptionkey@0"
I20260928 12:34:06.107733 24339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:06.120874 24339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:06.122673 24339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:06.122779 24339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:06.122953 24339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:06.123308 24365 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 12:34:06.124604 24365 ts_tablet_manager.cc:1852] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 12:34:06.124861 24365 ts_tablet_manager.cc:1905] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:06.127108 24365 ts_tablet_manager.cc:1918] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:06.127183 24365 log.cc:1196] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/d0ceb777aa3c4b5897d20684b1fe262b
I20260928 12:34:06.127429 24365 ts_tablet_manager.cc:1939] T d0ceb777aa3c4b5897d20684b1fe262b P 158ce3394b99467ca77a863028408fb5: Deleting consensus metadata
I20260928 12:34:06.127624 24339 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 12:34:06.127667 24339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 12:34:06.127693 24339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:06.127703 24339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:06.132990 24339 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:42285
I20260928 12:34:06.133101 24472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:42285 every 8 connection(s)
I20260928 12:34:06.133334 24339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:34:06 dist-test-slave-g31s krb5kdc[23570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598846, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:06.142038 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 24339
I20260928 12:34:06.142349 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:33681
--local_ip_for_outbound_sockets=127.8.13.3
--tserver_master_addrs=127.8.13.62:37829
--webserver_port=40865
--webserver_interface=127.8.13.3
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:34:06.142472 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.134019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:56559 (local address 127.8.13.62:37829)
0928 12:34:06.134077 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.134079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.134293 (+ 214us) server_negotiation.cc:407] Connection header received
0928 12:34:06.134865 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.134866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.134884 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.134931 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:06.135588 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.136110 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.137154 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.137413 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.137472 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.139822 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:06.139824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:06.139824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:06.139832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:06.141135 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.141520 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.141521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.141521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.141577 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.141824 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.141826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.141826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.141949 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:06.142006 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:06.142277 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 12:34:06.142378 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:34:06.142683 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.134019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:56559)
0928 12:34:06.134181 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:06.134215 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:06.134759 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:06.135014 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:06.135018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:06.135285 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:06.135488 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.135493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.136197 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.136199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.137030 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.137033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.137490 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.137491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.137626 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.138199 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:06.138208 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:06.139714 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:06.141216 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:06.141224 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:06.141227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:06.141451 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:06.141630 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:06.141631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:06.141631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:06.141738 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:06.142003 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:06.142011 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:06.142156 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 12:34:06.142257 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 12:34:06.142425 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260928 12:34:06.143116 24473 heartbeater.cc:347] Connected to a master server at 127.8.13.62:37829
I20260928 12:34:06.143218 24473 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:06.143404 24473 heartbeater.cc:511] Master 127.8.13.62:37829 requested a full tablet report, sending...
I20260928 12:34:06.143798 24146 ts_manager.cc:194] Registered new tserver with Master: 158ce3394b99467ca77a863028408fb5 (127.8.13.2:42285)
I20260928 12:34:06.144302 24146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:56559
W20260928 12:34:06.225688 24476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:06.225926 24476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:06.225952 24476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:06.227137 24476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:06.227177 24476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:06.227202 24476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:34:06.228587 24476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:33681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=40865
--tserver_master_addrs=127.8.13.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:06.228772 24476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:06.228940 24476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:06.233701 24481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:06.233867 24476 server_base.cc:1034] running on GCE node
W20260928 12:34:06.233681 24482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:06.233685 24484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:06.234182 24476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:06.234432 24476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:06.235602 24476 hybrid_clock.cc:648] HybridClock initialized: now 1790598846235586 us; error 32 us; skew 500 ppm
Sep 28 12:34:06 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598846, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:06.238336 24476 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:34:06.238855 24476 webserver.cc:460] Webserver started at http://127.8.13.3:40865/ using document root <none> and password file <none>
I20260928 12:34:06.239089 24476 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:06.239174 24476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:06.240581 24476 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:06.241139 24491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:06.241303 24476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:06.241386 24476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "70f6e5cb62ec47f1846df79d0a91c847"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "e4276a5bdea25ea14a92b22c5aad15ce"
server_key_iv: "d6307a630a01600b102c7a689841b76f"
server_key_version: "encryptionkey@0"
I20260928 12:34:06.241741 24476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:06.270583 24476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:06.272662 24476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:06.272804 24476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:06.273036 24476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:06.273358 24476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:06.273391 24476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:06.273422 24476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:06.273445 24476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:06.279222 24476 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:33681
I20260928 12:34:06.279271 24604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:33681 every 8 connection(s)
I20260928 12:34:06.279627 24476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:34:06 dist-test-slave-g31s krb5kdc[23570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598846, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:06.287880 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 24476
I20260928 12:34:06.289392 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.280361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:33415 (local address 127.8.13.62:37829)
0928 12:34:06.280438 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.280440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.280645 (+ 205us) server_negotiation.cc:407] Connection header received
0928 12:34:06.281224 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.281226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.281238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.281289 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:06.282295 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.282805 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.283837 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.284073 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.284129 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.286483 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:06.286485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:06.286485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:06.286494 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:06.287812 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.288275 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.288277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.288277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.288342 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.288636 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.288637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.288638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.288798 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:06.288873 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:06.289246 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 12:34:06.289324 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:34:06.289644 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.280355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:33415)
0928 12:34:06.280516 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:06.280560 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:06.281078 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:06.281351 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:06.281355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:06.281967 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:06.282181 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.282186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.282895 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.282896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.283753 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.283755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.284152 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.284153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.284264 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.284798 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:06.284808 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:06.286349 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:06.287913 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:06.287924 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:06.287926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:06.288201 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:06.288416 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:06.288416 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:06.288417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:06.288561 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:06.288895 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:06.288901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:06.289104 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 12:34:06.289202 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 12:34:06.289392 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260928 12:34:06.290163 24605 heartbeater.cc:347] Connected to a master server at 127.8.13.62:37829
I20260928 12:34:06.290264 24605 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:06.290448 24605 heartbeater.cc:511] Master 127.8.13.62:37829 requested a full tablet report, sending...
I20260928 12:34:06.290802 24146 ts_manager.cc:194] Registered new tserver with Master: 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681)
I20260928 12:34:06.291157 24146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:33415
I20260928 12:34:06.294795 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.288296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.8.13.62:37829)
0928 12:34:06.288428 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.288430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.288515 (+ 85us) server_negotiation.cc:407] Connection header received
0928 12:34:06.288552 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.288553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.288565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.288626 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:06.289256 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.290108 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.290918 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.291122 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.291177 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.292371 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:06.292373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:06.292373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:06.292380 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:06.293686 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.294041 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.294042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.294042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.294096 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.294259 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.294260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.294261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.294373 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:06.294573 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:06.294609 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 12:34:06.294692 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260928 12:34:06.295503 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:34:06.298367 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.295862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.8.13.62:37829)
0928 12:34:06.295923 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.295925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.296005 (+ 80us) server_negotiation.cc:407] Connection header received
0928 12:34:06.296064 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.296064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.296096 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.296162 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:34:06.296245 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.296723 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.297503 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.297735 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.297830 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.297974 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:34:06.298033 (+ 59us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 12:34:06.298296 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40238: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 12:34:06.298455 24079 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.13.62:37829)
W20260928 12:34:06.298501 24608 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40238: token expired
I20260928 12:34:06.304585 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.298696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.8.13.62:37829)
0928 12:34:06.298735 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.298737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.298868 (+ 131us) server_negotiation.cc:407] Connection header received
0928 12:34:06.298928 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.298929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.298940 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.298996 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:06.299560 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.300240 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.301058 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.301249 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.301301 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.302455 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:06.302456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:06.302457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:06.302464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:06.303658 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.303957 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.303957 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.303958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.304052 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:06.304180 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:06.304181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:06.304181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:06.304276 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:06.304348 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:06.304428 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 12:34:06.304486 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:34:06.305728 24146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:34:06.307385 24146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:34:06.309312 24265 tablet_service.cc:1467] Processing CreateTablet for tablet d4c574a05f454cc0bf95c20b7562ec4c (DEFAULT_TABLE table=test-table [id=eef649b567ae46b7931e66429cb0c8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:06.309509 24265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c574a05f454cc0bf95c20b7562ec4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:06.311038 24226 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: Bootstrap starting.
I20260928 12:34:06.311378 24226 tablet_bootstrap.cc:654] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:06.312139 24226 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: No bootstrap required, opened a new log
I20260928 12:34:06.312183 24226 ts_tablet_manager.cc:1397] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:34:06.312310 24226 raft_consensus.cc:348] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:06.312366 24226 raft_consensus.cc:374] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:06.312423 24226 raft_consensus.cc:729] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e95c13fe1d4128add928e345ef9565, State: Initialized, Role: FOLLOWER
I20260928 12:34:06.312472 24226 consensus_queue.cc:260] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:06.312561 24226 ts_tablet_manager.cc:1428] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:06.313982 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.309087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:33681 (local address 127.0.0.1:46308)
0928 12:34:06.309866 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:06.309875 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:06.309918 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:06.310099 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:06.310100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:06.310102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:06.310143 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.310144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.310985 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.310986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.312842 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.312844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.313748 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.313749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.313831 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.313835 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:34:06.313885 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:34:06.313914 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":948,"thread_start_us":101,"threads_started":1}
I20260928 12:34:06.313989 24613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.309203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.8.13.2:42285)
0928 12:34:06.309518 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.309522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.309533 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:34:06.309572 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.309574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.309590 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.309667 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:06.309778 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.310866 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.312796 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.313606 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.313682 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.313745 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:06.313867 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 12:34:06.313909 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":70,"threads_started":1}
I20260928 12:34:06.314002 24614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.309318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46308 (local address 127.8.13.3:33681)
0928 12:34:06.309481 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.309486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.309928 (+ 442us) server_negotiation.cc:407] Connection header received
0928 12:34:06.310016 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.310018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.310032 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.310098 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:06.310191 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.310905 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.312914 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.313682 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.313742 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.313788 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:06.313897 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 12:34:06.313932 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260928 12:34:06.313982 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.309052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:42285 (local address 127.0.0.1:48344)
0928 12:34:06.309129 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:06.309140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:06.309175 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:06.309690 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:06.309690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:06.309692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:06.309731 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.309733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.310957 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.310958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.311966 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.311969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.313679 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.313680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.313739 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.313761 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 12:34:06.313850 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:34:06.313896 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":39}
I20260928 12:34:06.314801 24407 tablet_service.cc:1467] Processing CreateTablet for tablet d4c574a05f454cc0bf95c20b7562ec4c (DEFAULT_TABLE table=test-table [id=eef649b567ae46b7931e66429cb0c8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:06.315044 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c574a05f454cc0bf95c20b7562ec4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:06.316665 24365 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Bootstrap starting.
I20260928 12:34:06.316974 24539 tablet_service.cc:1467] Processing CreateTablet for tablet d4c574a05f454cc0bf95c20b7562ec4c (DEFAULT_TABLE table=test-table [id=eef649b567ae46b7931e66429cb0c8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:06.317212 24539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c574a05f454cc0bf95c20b7562ec4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:06.317330 24365 tablet_bootstrap.cc:654] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:06.317572 24365 log.cc:824] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:06.318254 24365 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: No bootstrap required, opened a new log
I20260928 12:34:06.318310 24365 ts_tablet_manager.cc:1397] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:34:06.319747 24617 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Bootstrap starting.
I20260928 12:34:06.319672 24365 raft_consensus.cc:348] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:06.319826 24365 raft_consensus.cc:374] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:06.319856 24365 raft_consensus.cc:729] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158ce3394b99467ca77a863028408fb5, State: Initialized, Role: FOLLOWER
I20260928 12:34:06.319969 24365 consensus_queue.cc:260] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:06.320238 24365 ts_tablet_manager.cc:1428] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:34:06.320240 24473 heartbeater.cc:503] Master 127.8.13.62:37829 was elected leader, sending a full tablet report...
I20260928 12:34:06.320356 24617 tablet_bootstrap.cc:654] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:06.320873 24617 log.cc:824] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:06.321527 24617 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: No bootstrap required, opened a new log
I20260928 12:34:06.321583 24617 ts_tablet_manager.cc:1397] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:34:06.322990 24617 raft_consensus.cc:348] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:06.323134 24617 raft_consensus.cc:374] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:06.323155 24617 raft_consensus.cc:729] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f6e5cb62ec47f1846df79d0a91c847, State: Initialized, Role: FOLLOWER
I20260928 12:34:06.323252 24617 consensus_queue.cc:260] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:06.323477 24617 ts_tablet_manager.cc:1428] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:34:06.323549 24605 heartbeater.cc:503] Master 127.8.13.62:37829 was elected leader, sending a full tablet report...
W20260928 12:34:06.383882 24474 tablet.cc:2369] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:34:06.503641 24335 tablet.cc:2369] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:34:06.505584 24618 raft_consensus.cc:482] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:34:06.505688 24618 raft_consensus.cc:504] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:06.506033 24618 leader_election.cc:288] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681), 27e95c13fe1d4128add928e345ef9565 (127.8.13.1:46363)
I20260928 12:34:06.509982 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.506230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:33681 (local address 127.8.13.2:46963)
0928 12:34:06.506412 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:06.506424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:06.506484 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:06.506656 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:06.506657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:06.506660 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:06.506735 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.506736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.507629 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.507630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.508863 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.508867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.509689 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.509690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.509778 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.509782 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:34:06.509855 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 12:34:06.509894 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":2}
I20260928 12:34:06.510002 24614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.506352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:46963 (local address 127.8.13.3:33681)
0928 12:34:06.506398 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.506400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.506471 (+ 71us) server_negotiation.cc:407] Connection header received
0928 12:34:06.506573 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.506574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.506586 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.506638 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:06.506795 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.507533 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.508986 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.509585 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.509685 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.509698 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:06.509890 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 12:34:06.509934 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:34:06.510003 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.506329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:49437 (local address 127.8.13.1:46363)
0928 12:34:06.506377 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.506379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.506677 (+ 298us) server_negotiation.cc:407] Connection header received
0928 12:34:06.506709 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.506710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.506732 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.506787 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:06.506897 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.507656 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.508984 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.509676 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.509736 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.509836 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:06.509895 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 12:34:06.509937 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:34:06.509990 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.506230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:46363 (local address 127.8.13.2:49437)
0928 12:34:06.506606 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:06.506613 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:06.506667 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:06.506794 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:06.506794 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:06.506796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:06.506844 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.506845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.507720 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.507721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.508863 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:06.508867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.509739 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.509740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:06.509818 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.509821 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:06.509883 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:34:06.509911 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":177,"thread_start_us":137,"threads_started":1}
I20260928 12:34:06.510373 24559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c574a05f454cc0bf95c20b7562ec4c" candidate_uuid: "158ce3394b99467ca77a863028408fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f6e5cb62ec47f1846df79d0a91c847" is_pre_election: true
I20260928 12:34:06.510373 24287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c574a05f454cc0bf95c20b7562ec4c" candidate_uuid: "158ce3394b99467ca77a863028408fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e95c13fe1d4128add928e345ef9565" is_pre_election: true
I20260928 12:34:06.510524 24559 raft_consensus.cc:2454] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158ce3394b99467ca77a863028408fb5 in term 0.
I20260928 12:34:06.510527 24287 raft_consensus.cc:2454] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158ce3394b99467ca77a863028408fb5 in term 0.
I20260928 12:34:06.510757 24360 leader_election.cc:302] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158ce3394b99467ca77a863028408fb5, 27e95c13fe1d4128add928e345ef9565; no voters:
I20260928 12:34:06.510869 24618 raft_consensus.cc:2781] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:34:06.510929 24618 raft_consensus.cc:482] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:34:06.510946 24618 raft_consensus.cc:3037] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:06.511588 24618 raft_consensus.cc:504] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:06.511713 24618 leader_election.cc:288] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 1 election: Requested vote from peers 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681), 27e95c13fe1d4128add928e345ef9565 (127.8.13.1:46363)
I20260928 12:34:06.511878 24559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c574a05f454cc0bf95c20b7562ec4c" candidate_uuid: "158ce3394b99467ca77a863028408fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f6e5cb62ec47f1846df79d0a91c847"
I20260928 12:34:06.511927 24287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c574a05f454cc0bf95c20b7562ec4c" candidate_uuid: "158ce3394b99467ca77a863028408fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e95c13fe1d4128add928e345ef9565"
I20260928 12:34:06.511971 24559 raft_consensus.cc:3037] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:06.511997 24287 raft_consensus.cc:3037] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:06.512804 24559 raft_consensus.cc:2454] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158ce3394b99467ca77a863028408fb5 in term 1.
I20260928 12:34:06.512804 24287 raft_consensus.cc:2454] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158ce3394b99467ca77a863028408fb5 in term 1.
I20260928 12:34:06.512969 24360 leader_election.cc:302] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158ce3394b99467ca77a863028408fb5, 27e95c13fe1d4128add928e345ef9565; no voters:
I20260928 12:34:06.513046 24618 raft_consensus.cc:2781] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:06.513192 24618 raft_consensus.cc:686] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 LEADER]: Becoming Leader. State: Replica: 158ce3394b99467ca77a863028408fb5, State: Running, Role: LEADER
I20260928 12:34:06.513273 24618 consensus_queue.cc:237] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:06.514065 24146 catalog_manager.cc:5495] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 158ce3394b99467ca77a863028408fb5 (127.8.13.2). New cstate: current_term: 1 leader_uuid: "158ce3394b99467ca77a863028408fb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } health_report { overall_health: UNKNOWN } } }
W20260928 12:34:06.530158 24606 tablet.cc:2369] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:34:06.540380 24613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.538073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.8.13.2:42285)
0928 12:34:06.538126 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:06.538128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:06.538162 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:34:06.538198 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:06.538199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:06.538218 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:06.538269 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:34:06.538382 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.538919 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.539603 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:06.539792 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:06.539838 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:06.540067 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:34:06.540175 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:34:06.540244 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:06.540278 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 12:34:06.540297 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:34:06.542258 24559 raft_consensus.cc:1261] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 1 FOLLOWER]: Refusing update from remote peer 158ce3394b99467ca77a863028408fb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:06.542258 24287 raft_consensus.cc:1261] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Refusing update from remote peer 158ce3394b99467ca77a863028408fb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:06.542608 24622 consensus_queue.cc:1035] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:06.542742 24618 consensus_queue.cc:1035] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:06.545502 24627 mvcc.cc:205] Tried to move back new op lower bound from 7334292875434262528 to 7334292875318861824. Current Snapshot: MvccSnapshot[applied={T|T < 7334292875434262528}]
I20260928 12:34:06.547569 24145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:06.547663 24145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:06.548398 24145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Sending DeleteTablet for 3 replicas of tablet d4c574a05f454cc0bf95c20b7562ec4c
I20260928 12:34:06.548679 24265 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c574a05f454cc0bf95c20b7562ec4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:06 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:39598
I20260928 12:34:06.548708 24407 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c574a05f454cc0bf95c20b7562ec4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:06 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:48344
I20260928 12:34:06.548825 24539 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c574a05f454cc0bf95c20b7562ec4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:06 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:46308
I20260928 12:34:06.548928 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 24184
I20260928 12:34:06.549037 24634 tablet_replica.cc:330] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: stopping tablet replica
I20260928 12:34:06.549093 24634 raft_consensus.cc:2229] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:34:06.549136 24634 raft_consensus.cc:2258] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:06.549678 24636 tablet_replica.cc:330] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: stopping tablet replica
I20260928 12:34:06.549788 24636 raft_consensus.cc:2229] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:34:06.549937 24636 raft_consensus.cc:2258] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:06.550390 24636 ts_tablet_manager.cc:1905] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:06.552152 24636 ts_tablet_manager.cc:1918] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:06.552218 24636 log.cc:1196] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/d4c574a05f454cc0bf95c20b7562ec4c
I20260928 12:34:06.552470 24636 ts_tablet_manager.cc:1939] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Deleting consensus metadata
I20260928 12:34:06.552930 24130 catalog_manager.cc:4841] TS 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681): tablet d4c574a05f454cc0bf95c20b7562ec4c (table test-table [id=eef649b567ae46b7931e66429cb0c8c7]) successfully deleted
I20260928 12:34:06.554020 24635 tablet_replica.cc:330] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: stopping tablet replica
W20260928 12:34:06.554093 24131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:46363 (error 108)
I20260928 12:34:06.554174 24635 raft_consensus.cc:2229] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:34:06.554349 24635 raft_consensus.cc:2258] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 12:34:06.554714 24341 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:34:06.554498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:46363 (local address 127.0.0.1:39614)
0928 12:34:06.554632 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:06.554680 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:46363: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 12:34:06.554800 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 24339
I20260928 12:34:06.555050 24635 ts_tablet_manager.cc:1905] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 12:34:06.554864 24131 catalog_manager.cc:4584] TS 27e95c13fe1d4128add928e345ef9565 (127.8.13.1:46363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4c574a05f454cc0bf95c20b7562ec4c: Network error: Client connection negotiation failed: client connection to 127.8.13.1:46363: connect: Connection reset by peer (error 104)
I20260928 12:34:06.559430 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 24476
I20260928 12:34:06.559547 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:06.559424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:42285 (local address 127.0.0.1:48356)
0928 12:34:06.559476 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:06.559510 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.2:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:34:06.563952 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 24113
I20260928 12:34:26.569067 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:37829
--webserver_interface=127.8.13.62
--webserver_port=45111
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:37829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:26.650249 24638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:26.650470 24638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:26.650493 24638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:26.651695 24638 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 12:34:26.651738 24638 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 12:34:26.651750 24638 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 12:34:26.651760 24638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:26.651772 24638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:26.651784 24638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:26.651794 24638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:26.651809 24638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:26.653352 24638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:37829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:37829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:26.653595 24638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:26.653844 24638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:26.659149 24643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:26.659159 24638 server_base.cc:1034] running on GCE node
W20260928 12:34:26.659123 24644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:26.659106 24646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:26.659572 24638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:26.659893 24638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:26.661058 24638 hybrid_clock.cc:648] HybridClock initialized: now 1790598866661033 us; error 43 us; skew 500 ppm
Sep 28 12:34:26 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598866, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:26.663733 24638 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:34:26.664240 24638 webserver.cc:460] Webserver started at http://127.8.13.62:45111/ using document root <none> and password file <none>
I20260928 12:34:26.664445 24638 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:26.664491 24638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:26.665892 24638 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:26.666697 24653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:26.666855 24638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:26.666953 24638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "9a6041e25b004635af90249690bc5317"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "fe5a972f34cb11eb54368fc763984ce1"
server_key_iv: "957ace7f19ec3f0f1d8fc885e339a898"
server_key_version: "encryptionkey@0"
I20260928 12:34:26.667340 24638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:26.675832 24638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:26.677626 24638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:26.677749 24638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:26.681545 24638 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:37829
I20260928 12:34:26.681592 24705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:37829 every 8 connection(s)
I20260928 12:34:26.681993 24638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:26.683485 24706 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:34:26.683888 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 24638
I20260928 12:34:26.684136 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:46363
--local_ip_for_outbound_sockets=127.8.13.1
--tserver_master_addrs=127.8.13.62:37829
--webserver_port=44457
--webserver_interface=127.8.13.1
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:34:26.684157 24706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Bootstrap starting.
I20260928 12:34:26.692535 24706 log.cc:824] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:26.695007 24706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:34:26.695326 24706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Bootstrap complete.
I20260928 12:34:26.697433 24706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:34:26.697680 24706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a6041e25b004635af90249690bc5317, State: Initialized, Role: FOLLOWER
I20260928 12:34:26.697854 24706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:34:26.697938 24706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:26.697989 24706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:26.698038 24706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 2 FOLLOWER]: Advancing to term 3
I20260928 12:34:26.699153 24706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:34:26.699250 24706 leader_election.cc:302] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a6041e25b004635af90249690bc5317; no voters:
I20260928 12:34:26.699432 24706 leader_election.cc:288] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 12:34:26.699491 24710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 12:34:26.699615 24710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [term 3 LEADER]: Becoming Leader. State: Replica: 9a6041e25b004635af90249690bc5317, State: Running, Role: LEADER
I20260928 12:34:26.699723 24706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:26.699767 24710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } }
I20260928 12:34:26.700225 24712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9a6041e25b004635af90249690bc5317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } } }
I20260928 12:34:26.700314 24712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:26.700591 24719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:26.700666 24710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a6041e25b004635af90249690bc5317. Latest consensus state: current_term: 3 leader_uuid: "9a6041e25b004635af90249690bc5317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6041e25b004635af90249690bc5317" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 37829 } } }
I20260928 12:34:26.700730 24710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:26.701547 24719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:26.701675 24719 catalog_manager.cc:1259] Loaded cluster ID: 2b971c0008654fd09daa16029687aff6
I20260928 12:34:26.701742 24719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:26.703090 24719 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:26.703253 24719 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Loaded TSK: 0
I20260928 12:34:26.703290 24719 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a6041e25b004635af90249690bc5317: Loaded TSK: 1
F20260928 12:34:26.703508 24719 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4e73c1c9ad google::LogMessage::Fail() at ??:0
@ 0x7f4e73c1f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4e73c1c4de google::LogMessage::Flush() at ??:0
@ 0x7f4e73c1da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4e76389104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4e76386d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4e7bb761cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4e7bb7648e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4e7bb76594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4e7bb3cc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f4e7bb46ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4e75aea0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4e75adf954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4e769436db start_thread at ??:0
@ 0x7f4e7271a61f clone at ??:0
W20260928 12:34:26.780961 24708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:26.781191 24708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:26.781219 24708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:26.782577 24708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:26.782636 24708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:26.782667 24708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:34:26.784279 24708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:46363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=44457
--tserver_master_addrs=127.8.13.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:26.784535 24708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:26.784772 24708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:26.789891 24748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:26.789891 24747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:26.789894 24750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:26.790447 24708 server_base.cc:1034] running on GCE node
I20260928 12:34:26.790661 24708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:26.790905 24708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:26.792065 24708 hybrid_clock.cc:648] HybridClock initialized: now 1790598866792049 us; error 31 us; skew 500 ppm
Sep 28 12:34:26 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598866, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:26.794622 24708 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:34:26.795081 24708 webserver.cc:460] Webserver started at http://127.8.13.1:44457/ using document root <none> and password file <none>
I20260928 12:34:26.795307 24708 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:26.795353 24708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:26.796783 24708 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:34:26.797557 24757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:26.797712 24708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:34:26.797819 24708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "27e95c13fe1d4128add928e345ef9565"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "9cb264776e9d19f35ad2523aec4a4029"
server_key_iv: "3ee1eeb6065fc75fe5b138975739b392"
server_key_version: "encryptionkey@0"
I20260928 12:34:26.798195 24708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:26.817582 24708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:26.819710 24708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:26.819842 24708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:26.820050 24708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:26.820463 24764 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:34:26.822345 24708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:34:26.822396 24708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:34:26.822449 24708 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:34:26.823041 24708 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:34:26.823074 24708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:34:26.823159 24764 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: Bootstrap starting.
I20260928 12:34:26.829370 24708 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:46363
I20260928 12:34:26.829727 24708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:34:26.830241 24871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:46363 every 8 connection(s)
I20260928 12:34:26.833712 24764 log.cc:824] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:26.836561 24764 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:34:26.836894 24764 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: Bootstrap complete.
I20260928 12:34:26.838022 24764 ts_tablet_manager.cc:1397] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.007s
I20260928 12:34:26.839166 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 24708
I20260928 12:34:26.839419 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:42285
--local_ip_for_outbound_sockets=127.8.13.2
--tserver_master_addrs=127.8.13.62:37829
--webserver_port=33289
--webserver_interface=127.8.13.2
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:34:26.839530 24764 raft_consensus.cc:348] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:26.839886 24764 raft_consensus.cc:729] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e95c13fe1d4128add928e345ef9565, State: Initialized, Role: FOLLOWER
I20260928 12:34:26.840046 24764 consensus_queue.cc:260] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:26.840334 24764 ts_tablet_manager.cc:1428] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260928 12:34:26.898851 24874 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:34:26.830374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:59873)
0928 12:34:26.830538 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:26.830579 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:26.831011 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:26.898616 (+ 67605us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
W20260928 12:34:26.899103 24872 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:37829 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.62:37829: Client connection negotiation failed: client connection to 127.8.13.62:37829: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:34:26.899418 24874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:26.899304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:56815)
0928 12:34:26.899350 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:26.899389 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 12:34:26.899703 24874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:26.899630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:37677)
0928 12:34:26.899667 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:26.899688 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 12:34:26.899816 24872 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:34:26.921255 24877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:26.921454 24877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:26.921476 24877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:26.922647 24877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:26.922688 24877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:26.922713 24877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:34:26.924113 24877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:42285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=33289
--tserver_master_addrs=127.8.13.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:26.924319 24877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:26.924520 24877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:26.929344 24882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:26.929315 24883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:26.929374 24877 server_base.cc:1034] running on GCE node
W20260928 12:34:26.929315 24885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:26.929749 24877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:26.929992 24877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:26.931151 24877 hybrid_clock.cc:648] HybridClock initialized: now 1790598866931138 us; error 27 us; skew 500 ppm
Sep 28 12:34:26 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598866, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:26.933547 24877 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:34:26.933997 24877 webserver.cc:460] Webserver started at http://127.8.13.2:33289/ using document root <none> and password file <none>
I20260928 12:34:26.934211 24877 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:26.934259 24877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:26.935596 24877 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:26.936352 24892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:26.936532 24877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:34:26.936625 24877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "158ce3394b99467ca77a863028408fb5"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "eadeb82be1e22382aa49e0c78844534d"
server_key_iv: "4aea462aaec4d0e3a556fd4f37f6cebb"
server_key_version: "encryptionkey@0"
I20260928 12:34:26.936996 24877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:26.951355 24877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:26.953320 24877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:26.953476 24877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:26.953717 24877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:26.954183 24899 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:34:26.955694 24877 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:34:26.955745 24877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:34:26.955780 24877 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:34:26.956475 24877 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:34:26.956509 24877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:34:26.956593 24899 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Bootstrap starting.
I20260928 12:34:26.958398 24899 log.cc:824] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:26.961472 24899 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:34:26.961825 24899 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Bootstrap complete.
I20260928 12:34:26.961987 24899 ts_tablet_manager.cc:1397] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 12:34:26.962540 24877 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:42285
I20260928 12:34:26.962611 25007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:42285 every 8 connection(s)
I20260928 12:34:26.962879 24877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:34:26.963620 24899 raft_consensus.cc:348] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:26.963871 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:26.963491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:42803)
0928 12:34:26.963667 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:26.963747 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260928 12:34:26.963876 24899 raft_consensus.cc:729] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158ce3394b99467ca77a863028408fb5, State: Initialized, Role: FOLLOWER
W20260928 12:34:26.964097 25008 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:37829 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.62:37829: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
I20260928 12:34:26.964147 24899 consensus_queue.cc:260] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:26.964370 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:26.964269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:49137)
0928 12:34:26.964308 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:26.964339 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 12:34:26.964398 24899 ts_tablet_manager.cc:1428] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:34:26.964597 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 24877
I20260928 12:34:26.964674 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:26.964578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:52637)
0928 12:34:26.964625 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:26.964653 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 12:34:26.964792 25008 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:34:26.964862 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:33681
--local_ip_for_outbound_sockets=127.8.13.3
--tserver_master_addrs=127.8.13.62:37829
--webserver_port=40865
--webserver_interface=127.8.13.3
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:27.044631 25012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:27.044875 25012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:27.044902 25012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:27.046151 25012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:27.046200 25012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:27.046229 25012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:34:27.047720 25012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:33681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=40865
--tserver_master_addrs=127.8.13.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:27.047962 25012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:27.048195 25012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:27.053171 25020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:27.053292 25012 server_base.cc:1034] running on GCE node
W20260928 12:34:27.053175 25018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:27.053259 25017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:27.053606 25012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:27.053862 25012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:27.055006 25012 hybrid_clock.cc:648] HybridClock initialized: now 1790598867054993 us; error 31 us; skew 500 ppm
Sep 28 12:34:27 dist-test-slave-g31s krb5kdc[23570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598867, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:27.057446 25012 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:34:27.057905 25012 webserver.cc:460] Webserver started at http://127.8.13.3:40865/ using document root <none> and password file <none>
I20260928 12:34:27.058107 25012 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:27.058151 25012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:27.059499 25012 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:34:27.060071 25027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:27.060204 25012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:34:27.060286 25012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "70f6e5cb62ec47f1846df79d0a91c847"
format_stamp: "Formatted at 2026-09-28 12:33:45 on dist-test-slave-g31s"
server_key: "e4276a5bdea25ea14a92b22c5aad15ce"
server_key_iv: "d6307a630a01600b102c7a689841b76f"
server_key_version: "encryptionkey@0"
I20260928 12:34:27.060626 25012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:27.079833 25012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:27.081929 25012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:27.082086 25012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:27.082324 25012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:27.082648 25012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:27.082682 25012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:27.082715 25012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:27.082737 25012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:27.088222 25012 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:33681
I20260928 12:34:27.088285 25140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:33681 every 8 connection(s)
I20260928 12:34:27.088630 25012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:34:27.089540 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 25012
I20260928 12:34:27.089740 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:27.089320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:52077)
0928 12:34:27.089487 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:27.089567 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
W20260928 12:34:27.090181 25141 heartbeater.cc:644] Failed to heartbeat to 127.8.13.62:37829 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.62:37829: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
I20260928 12:34:27.090518 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:27.090379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:51625)
0928 12:34:27.090472 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:27.090497 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 12:34:27.090766 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:27.090695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:37403)
0928 12:34:27.090735 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:27.090753 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:34:27.090870 25141 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:34:27.900553 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:27.900238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:53419)
0928 12:34:27.900433 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:27.900484 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
I20260928 12:34:27.965381 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:27.965167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:45075)
0928 12:34:27.965284 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:27.965326 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260928 12:34:28.091449 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:28.091237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:34469)
0928 12:34:28.091358 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:28.091392 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260928 12:34:28.209550 25148 raft_consensus.cc:482] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:34:28.209666 25148 raft_consensus.cc:504] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:28.210022 25148 leader_election.cc:288] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681), 27e95c13fe1d4128add928e345ef9565 (127.8.13.1:46363)
Sep 28 12:34:28 dist-test-slave-g31s krb5kdc[23570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598866, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.3@KRBTEST.COM
Sep 28 12:34:28 dist-test-slave-g31s krb5kdc[23570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598866, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.1@KRBTEST.COM
I20260928 12:34:28.225730 25150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:28.210276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:48743 (local address 127.8.13.3:33681)
0928 12:34:28.210453 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:28.210465 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:28.210494 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:34:28.211017 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:28.211020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:28.211055 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:28.211149 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:28.211815 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:28.212971 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:28.214016 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:28.214639 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:28.214752 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:28.222641 (+ 7889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:28.222647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:28.222647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:28.222660 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:28.224214 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:28.224618 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:28.224621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:28.224622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:28.224711 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:28.224932 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:28.224932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:28.224933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:28.225165 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:28.225231 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:28.225506 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 12:34:28.225621 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260928 12:34:28.225730 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:28.210298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:34337 (local address 127.8.13.1:46363)
0928 12:34:28.210543 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:28.210547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:28.210555 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:34:28.210615 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:28.210618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:28.210639 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:28.210717 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:28.211405 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:28.212655 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:28.214017 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:28.214642 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:28.214752 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:28.222896 (+ 8144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:28.222900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:28.222900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:28.222910 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:28.224346 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:28.224675 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:28.224678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:28.224678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:28.224759 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:28.224939 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:28.224940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:28.224940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:28.225168 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:28.225235 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:28.225505 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 12:34:28.225628 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":66,"threads_started":1}
I20260928 12:34:28.225749 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:28.210177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:46363 (local address 127.8.13.2:34337)
0928 12:34:28.210296 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:28.210305 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:28.210419 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:28.210759 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:28.210762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:28.211042 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:28.211294 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:28.211299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:28.212767 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:28.212768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:28.213937 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:28.213940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:28.214729 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:28.214729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:28.214930 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:28.220579 (+ 5649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:28.220589 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:28.222768 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:28.224429 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:28.224430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:28.224431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:28.224608 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:28.224816 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:28.224817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:28.224817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:28.224887 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:28.225229 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:28.225230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:28.225364 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 12:34:28.225448 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 12:34:28.225637 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":43,"thread_start_us":64,"threads_started":1}
I20260928 12:34:28.226107 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:28.210149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:33681 (local address 127.8.13.2:48743)
0928 12:34:28.210192 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:28.210219 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:28.210394 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:28.211160 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:28.211160 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:28.211647 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:28.211699 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:28.211701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:28.213067 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:28.213068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:28.213937 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:28.213940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:28.214727 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:28.214728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:28.214930 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:28.220576 (+ 5646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:28.220589 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:28.222485 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:28.224292 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:28.224299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:28.224301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:28.224548 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:28.224773 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:28.224774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:28.224774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:28.224879 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:28.225224 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:28.225230 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:28.225362 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 12:34:28.225448 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 12:34:28.225709 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":125}
W20260928 12:34:28.226918 24893 leader_election.cc:341] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681): Not found: Tablet not found: d4c574a05f454cc0bf95c20b7562ec4c
I20260928 12:34:28.227020 24826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c574a05f454cc0bf95c20b7562ec4c" candidate_uuid: "158ce3394b99467ca77a863028408fb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27e95c13fe1d4128add928e345ef9565" is_pre_election: true
I20260928 12:34:28.227167 24826 raft_consensus.cc:2454] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158ce3394b99467ca77a863028408fb5 in term 1.
I20260928 12:34:28.227423 24894 leader_election.cc:302] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 158ce3394b99467ca77a863028408fb5, 27e95c13fe1d4128add928e345ef9565; no voters: 70f6e5cb62ec47f1846df79d0a91c847
I20260928 12:34:28.227543 25148 raft_consensus.cc:2781] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 12:34:28.227603 25148 raft_consensus.cc:482] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:34:28.227626 25148 raft_consensus.cc:3037] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:34:28.228761 25148 raft_consensus.cc:504] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:28.228911 25148 leader_election.cc:288] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 2 election: Requested vote from peers 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681), 27e95c13fe1d4128add928e345ef9565 (127.8.13.1:46363)
I20260928 12:34:28.229063 24826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c574a05f454cc0bf95c20b7562ec4c" candidate_uuid: "158ce3394b99467ca77a863028408fb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27e95c13fe1d4128add928e345ef9565"
I20260928 12:34:28.229146 24826 raft_consensus.cc:3037] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 1 FOLLOWER]: Advancing to term 2
W20260928 12:34:28.229176 24893 leader_election.cc:341] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681): Not found: Tablet not found: d4c574a05f454cc0bf95c20b7562ec4c
I20260928 12:34:28.230188 24826 raft_consensus.cc:2454] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158ce3394b99467ca77a863028408fb5 in term 2.
I20260928 12:34:28.230336 24894 leader_election.cc:302] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 158ce3394b99467ca77a863028408fb5, 27e95c13fe1d4128add928e345ef9565; no voters: 70f6e5cb62ec47f1846df79d0a91c847
I20260928 12:34:28.230443 25148 raft_consensus.cc:2781] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:34:28.230580 25148 raft_consensus.cc:686] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [term 2 LEADER]: Becoming Leader. State: Replica: 158ce3394b99467ca77a863028408fb5, State: Running, Role: LEADER
I20260928 12:34:28.230669 25148 consensus_queue.cc:237] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:28.230860 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:28.230768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:39063)
0928 12:34:28.230813 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:28.230835 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:34:28.644775 24826 raft_consensus.cc:1261] T d4c574a05f454cc0bf95c20b7562ec4c P 27e95c13fe1d4128add928e345ef9565 [term 2 FOLLOWER]: Refusing update from remote peer 158ce3394b99467ca77a863028408fb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 12:34:28.645123 25148 consensus_queue.cc:1035] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 12:34:28.646073 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:28.645885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:47671)
0928 12:34:28.645967 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:28.646030 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":34,"threads_started":1}
I20260928 12:34:28.646889 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:28.646779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:41375)
0928 12:34:28.646825 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:28.646854 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:34:28.646940 24893 consensus_peers.cc:487] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5 -> Peer 70f6e5cb62ec47f1846df79d0a91c847 (127.8.13.3:33681): Couldn't send request to peer 70f6e5cb62ec47f1846df79d0a91c847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4c574a05f454cc0bf95c20b7562ec4c. This is attempt 1: this message will repeat every 5th retry.
I20260928 12:34:28.647080 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:28.646983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:57703)
0928 12:34:28.647026 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:28.647052 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 12:34:29.046995 25159 ts_tablet_manager.cc:927] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Initiating tablet copy from peer 158ce3394b99467ca77a863028408fb5 (127.8.13.2:42285)
I20260928 12:34:29.047209 25159 tablet_copy_client.cc:316] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: tablet copy: Beginning tablet copy session from remote peer at address 127.8.13.2:42285
Sep 28 12:34:29 dist-test-slave-g31s krb5kdc[23570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598867, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.2@KRBTEST.COM
I20260928 12:34:29.054962 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:29.047478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:42285 (local address 127.8.13.3:39339)
0928 12:34:29.047628 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:29.047649 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:29.047717 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:29.047869 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:29.047873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:29.048170 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:29.048235 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:29.048238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:29.049126 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:29.049127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:29.049748 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:29.049751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:29.050244 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:29.050245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:29.050330 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:29.050832 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:29.050837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:29.052232 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:29.053855 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:29.053862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:29.053864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:29.054064 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:29.054261 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:29.054262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:29.054262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:29.054327 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:29.054600 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:29.054608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:29.054664 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 12:34:29.054742 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 12:34:29.054845 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260928 12:34:29.054986 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:29.047580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:39339 (local address 127.8.13.2:42285)
0928 12:34:29.047695 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:29.047701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:29.047713 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:34:29.047772 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:29.047776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:29.047791 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:29.047854 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:29.048320 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:29.049043 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:29.049856 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:29.050174 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:29.050236 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:29.052340 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:29.052353 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:29.052353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:29.052360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:29.053716 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:29.054145 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:29.054148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:29.054148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:29.054201 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:29.054372 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:29.054373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:29.054373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:29.054524 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:29.054584 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:29.054807 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 12:34:29.054888 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":44,"threads_started":1}
I20260928 12:34:29.055426 24982 tablet_copy_service.cc:140] P 158ce3394b99467ca77a863028408fb5: Received BeginTabletCopySession request for tablet d4c574a05f454cc0bf95c20b7562ec4c from peer 70f6e5cb62ec47f1846df79d0a91c847 ({username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:39339)
I20260928 12:34:29.055491 24982 tablet_copy_service.cc:161] P 158ce3394b99467ca77a863028408fb5: Beginning new tablet copy session on tablet d4c574a05f454cc0bf95c20b7562ec4c from peer 70f6e5cb62ec47f1846df79d0a91c847 at {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:39339: session id = 70f6e5cb62ec47f1846df79d0a91c847-d4c574a05f454cc0bf95c20b7562ec4c
I20260928 12:34:29.055994 24982 tablet_copy_source_session.cc:205] T d4c574a05f454cc0bf95c20b7562ec4c P 158ce3394b99467ca77a863028408fb5: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 12:34:29.056885 25159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c574a05f454cc0bf95c20b7562ec4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:29.059532 25159 tablet_copy_client.cc:791] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: tablet copy: Starting download of 0 data blocks...
I20260928 12:34:29.059659 25159 tablet_copy_client.cc:655] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: tablet copy: Starting download of 1 WAL segments...
I20260928 12:34:29.060520 25159 tablet_copy_client.cc:531] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 12:34:29.061299 25159 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Bootstrap starting.
I20260928 12:34:29.062805 25159 log.cc:824] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:29.065462 25159 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:34:29.065733 25159 tablet_bootstrap.cc:492] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Bootstrap complete.
I20260928 12:34:29.065907 25159 ts_tablet_manager.cc:1397] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.003s
I20260928 12:34:29.066957 25159 raft_consensus.cc:348] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:29.067119 25159 raft_consensus.cc:729] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f6e5cb62ec47f1846df79d0a91c847, State: Initialized, Role: FOLLOWER
I20260928 12:34:29.067226 25159 consensus_queue.cc:260] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ce3394b99467ca77a863028408fb5" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42285 } } peers { permanent_uuid: "70f6e5cb62ec47f1846df79d0a91c847" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33681 } } peers { permanent_uuid: "27e95c13fe1d4128add928e345ef9565" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 46363 } }
I20260928 12:34:29.067507 25159 ts_tablet_manager.cc:1428] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 12:34:29.067829 24982 tablet_copy_service.cc:342] P 158ce3394b99467ca77a863028408fb5: Request end of tablet copy session 70f6e5cb62ec47f1846df79d0a91c847-d4c574a05f454cc0bf95c20b7562ec4c received from {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:39339
I20260928 12:34:29.067847 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:29.067726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:49313)
0928 12:34:29.067774 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:29.067815 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 12:34:29.067888 24982 tablet_copy_service.cc:433] P 158ce3394b99467ca77a863028408fb5: ending tablet copy session 70f6e5cb62ec47f1846df79d0a91c847-d4c574a05f454cc0bf95c20b7562ec4c on tablet d4c574a05f454cc0bf95c20b7562ec4c with peer 70f6e5cb62ec47f1846df79d0a91c847
I20260928 12:34:29.068192 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:29.068111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:41131)
0928 12:34:29.068147 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:29.068171 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 12:34:29.517067 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:29.516831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:54797)
0928 12:34:29.516950 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:29.517018 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260928 12:34:29.517016 25095 raft_consensus.cc:1203] T d4c574a05f454cc0bf95c20b7562ec4c P 70f6e5cb62ec47f1846df79d0a91c847 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 12:34:29.517477 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:29.517376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:48975)
0928 12:34:29.517421 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:29.517446 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:34:29.647701 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:29.647481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:42805)
0928 12:34:29.647608 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:29.647652 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260928 12:34:30.517704 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:30.517481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:48331)
0928 12:34:30.517620 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:30.517658 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260928 12:34:30.518090 25169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:30.517907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:52637)
0928 12:34:30.518010 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:30.518042 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
I20260928 12:34:30.648509 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:30.648257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:33177)
0928 12:34:30.648411 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:30.648457 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260928 12:34:31.518416 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:31.518172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:58261)
0928 12:34:31.518327 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:31.518371 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260928 12:34:31.518635 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:31.518464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:38317)
0928 12:34:31.518571 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:31.518597 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":48,"threads_started":1}
I20260928 12:34:31.649312 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:31.649062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:38477)
0928 12:34:31.649211 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:31.649260 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260928 12:34:32.519181 25175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:32.519012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:50625)
0928 12:34:32.519114 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:32.519142 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260928 12:34:32.519177 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:32.518952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:59945)
0928 12:34:32.519098 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:32.519132 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260928 12:34:32.519555 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:32.519467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:35461)
0928 12:34:32.519511 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:32.519535 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 12:34:32.650012 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:32.649781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:34027)
0928 12:34:32.649934 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:32.649965 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
I20260928 12:34:33.519816 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:33.519602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:36283)
0928 12:34:33.519720 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:33.519760 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260928 12:34:33.520080 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:33.519908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:58997)
0928 12:34:33.520015 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:33.520045 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260928 12:34:33.650756 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:33.650514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:36671)
0928 12:34:33.650658 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:33.650705 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260928 12:34:34.520423 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:34.520193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:53841)
0928 12:34:34.520340 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:34.520377 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260928 12:34:34.520645 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:34.520449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:39407)
0928 12:34:34.520578 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:34.520610 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260928 12:34:34.651544 25182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:34.651284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:42885)
0928 12:34:34.651444 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:34.651492 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260928 12:34:35.521255 25184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:35.520994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:36577)
0928 12:34:35.521168 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:35.521199 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260928 12:34:35.521255 25183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:35.521006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:44075)
0928 12:34:35.521161 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:35.521203 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260928 12:34:35.652313 25185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:35.652054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:55923)
0928 12:34:35.652218 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:35.652264 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260928 12:34:36.521965 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:36.521731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:55209)
0928 12:34:36.521881 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:36.521917 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260928 12:34:36.521965 25186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:36.521711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:48049)
0928 12:34:36.521884 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:36.521923 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":60,"threads_started":1}
I20260928 12:34:36.653080 25188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:36.652815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:52881)
0928 12:34:36.652980 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:36.653028 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260928 12:34:37.522693 25189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:37.522464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:46767)
0928 12:34:37.522599 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:37.522639 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260928 12:34:37.522693 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:37.522465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:35961)
0928 12:34:37.522611 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:37.522642 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260928 12:34:37.653892 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:37.653583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:46949)
0928 12:34:37.653748 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:37.653829 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260928 12:34:38.523422 25192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:38.523164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:37665)
0928 12:34:38.523308 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:38.523364 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260928 12:34:38.523519 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:38.523290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:55315)
0928 12:34:38.523427 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:38.523469 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260928 12:34:38.654642 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:38.654399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:45951)
0928 12:34:38.654543 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:38.654593 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260928 12:34:39.524273 25196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:39.524020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:40247)
0928 12:34:39.524178 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:39.524222 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260928 12:34:39.524274 25195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:39.524025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:32883)
0928 12:34:39.524177 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:39.524220 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260928 12:34:39.655460 25197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:39.655198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:48877)
0928 12:34:39.655365 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:39.655411 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
I20260928 12:34:40.524971 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:40.524741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:47357)
0928 12:34:40.524884 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:40.524916 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260928 12:34:40.524992 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:40.524741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:41779)
0928 12:34:40.524891 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:40.524944 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260928 12:34:40.656163 25200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:40.655938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:55283)
0928 12:34:40.656065 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:40.656110 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260928 12:34:41.525714 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:41.525488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.2:38839)
0928 12:34:41.525629 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:41.525667 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260928 12:34:41.525717 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:41.525532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.3:45949)
0928 12:34:41.525647 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:41.525677 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260928 12:34:41.656935 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:41.656644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:37829 (local address 127.8.13.1:57763)
0928 12:34:41.656826 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:41.656877 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.62:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":75,"threads_started":1}
I20260928 12:34:42.096094 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 24708
I20260928 12:34:42.100741 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 24877
I20260928 12:34:42.105105 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 25012
2026-09-28T12:34:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58198 ms)
[----------] 30 tests from SecurityITest (242091 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:42 dist-test-slave-g31s krb5kdc[25206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:42 dist-test-slave-g31s krb5kdc[25206](info): set up 2 sockets
Sep 28 12:34:42 dist-test-slave-g31s krb5kdc[25206](info): commencing operation
krb5kdc: starting...
W20260928 12:34:43.070156 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:34:43 dist-test-slave-g31s krb5kdc[25206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:34:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:34:43.184600 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:42601
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:37821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:42601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:43.264571 25219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:43.264791 25219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:43.264822 25219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:43.266033 25219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:43.266077 25219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:43.266095 25219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:43.266112 25219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:43.266126 25219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:43.267614 25219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:37821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:42601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:42601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:43.267903 25219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:43.268126 25219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:43.273375 25227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:43.273432 25219 server_base.cc:1034] running on GCE node
W20260928 12:34:43.273386 25225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:43.273391 25224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:43.273921 25219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:43.274221 25219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:43.275367 25219 hybrid_clock.cc:648] HybridClock initialized: now 1790598883275352 us; error 31 us; skew 500 ppm
Sep 28 12:34:43 dist-test-slave-g31s krb5kdc[25206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598883, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:43.277791 25219 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:34:43.278301 25219 webserver.cc:460] Webserver started at http://127.8.13.62:45741/ using document root <none> and password file <none>
I20260928 12:34:43.278514 25219 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:43.278563 25219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:43.278685 25219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:43.279732 25219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "69bc4d2ad40e4d93983d4160154f39da"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "bac0ac40c929272309a5a5ffdb7535fe"
server_key_iv: "8a9ffd17ff7a03217f5e0bbe60c4ff99"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.280089 25219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "69bc4d2ad40e4d93983d4160154f39da"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "bac0ac40c929272309a5a5ffdb7535fe"
server_key_iv: "8a9ffd17ff7a03217f5e0bbe60c4ff99"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.281353 25219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:34:43.282119 25234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:43.282290 25219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:43.282380 25219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "69bc4d2ad40e4d93983d4160154f39da"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "bac0ac40c929272309a5a5ffdb7535fe"
server_key_iv: "8a9ffd17ff7a03217f5e0bbe60c4ff99"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.282444 25219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:43.297377 25219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:43.299561 25219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:43.299708 25219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:43.303580 25219 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:42601
I20260928 12:34:43.303614 25286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:42601 every 8 connection(s)
I20260928 12:34:43.303951 25219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:43.304770 25287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:43.307054 25287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da: Bootstrap starting.
I20260928 12:34:43.307590 25287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:43.307838 25287 log.cc:824] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:43.308455 25287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da: No bootstrap required, opened a new log
I20260928 12:34:43.309196 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 25219
I20260928 12:34:43.309290 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:43.309482 8244 external_mini_cluster.cc:1428] Setting key 90ea866ae3030d09238f8fd5f15f1fd4
I20260928 12:34:43.310370 25287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bc4d2ad40e4d93983d4160154f39da" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42601 } }
I20260928 12:34:43.310534 25287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:43.310607 25287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69bc4d2ad40e4d93983d4160154f39da, State: Initialized, Role: FOLLOWER
I20260928 12:34:43.310752 25287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bc4d2ad40e4d93983d4160154f39da" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42601 } }
I20260928 12:34:43.310815 25287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:43.310835 25287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:43.310878 25287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:43.311882 25287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bc4d2ad40e4d93983d4160154f39da" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42601 } }
I20260928 12:34:43.312000 25287 leader_election.cc:302] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69bc4d2ad40e4d93983d4160154f39da; no voters:
I20260928 12:34:43.312160 25287 leader_election.cc:288] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:43.312218 25292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:43.312378 25292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [term 1 LEADER]: Becoming Leader. State: Replica: 69bc4d2ad40e4d93983d4160154f39da, State: Running, Role: LEADER
I20260928 12:34:43.312387 25287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:43.312546 25292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bc4d2ad40e4d93983d4160154f39da" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42601 } }
I20260928 12:34:43.313616 25293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69bc4d2ad40e4d93983d4160154f39da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bc4d2ad40e4d93983d4160154f39da" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42601 } } }
I20260928 12:34:43.313699 25293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:43.313874 25294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69bc4d2ad40e4d93983d4160154f39da. Latest consensus state: current_term: 1 leader_uuid: "69bc4d2ad40e4d93983d4160154f39da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bc4d2ad40e4d93983d4160154f39da" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 42601 } } }
I20260928 12:34:43.314204 25294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da [sys.catalog]: This master's current role is: LEADER
W20260928 12:34:43.314476 25307 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:34:43.314567 25307 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:34:43.314653 25308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:43.314814 25308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:43.316284 25308 catalog_manager.cc:1347] Generated new cluster ID: a228c8fe0e1c408098b51a1c872a5eb3
I20260928 12:34:43.316334 25308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:34:43 dist-test-slave-g31s krb5kdc[25206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:43.320621 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.309991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.8.13.62:42601)
0928 12:34:43.310183 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:43.310192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:43.310228 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:34:43.310343 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:43.310350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:43.310405 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:43.310558 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:43.311247 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.312616 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.313698 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.314315 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.314421 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.316992 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:43.316996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:43.316996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:43.317008 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:43.318920 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:43.319273 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:43.319276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:43.319277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:43.319406 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:43.319578 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:43.319578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:43.319579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:43.319917 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:43.320092 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:43.320160 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 12:34:43.320408 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260928 12:34:43.321895 25308 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:43.322364 25308 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:43.326154 25308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da: Generated new TSK 0
I20260928 12:34:43.326321 25308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:34:43.352567 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42601
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:37821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:43.434026 25313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:43.434230 25313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:43.434254 25313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:43.435634 25313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:43.435702 25313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:43.435742 25313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:34:43.437587 25313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:37821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:43.437893 25313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:43.438143 25313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:34:43.443372 25313 server_base.cc:1034] running on GCE node
W20260928 12:34:43.443336 25321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:43.443354 25318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:43.443326 25319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:43.443725 25313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:43.444010 25313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:43.445173 25313 hybrid_clock.cc:648] HybridClock initialized: now 1790598883445154 us; error 36 us; skew 500 ppm
Sep 28 12:34:43 dist-test-slave-g31s krb5kdc[25206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598883, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:43.447794 25313 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:34:43.448251 25313 webserver.cc:460] Webserver started at http://127.8.13.1:36423/ using document root <none> and password file <none>
I20260928 12:34:43.448486 25313 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:43.448539 25313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:43.448664 25313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:43.449769 25313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "d84a54a8f8ee4879bf7c3463fd17adf2"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "9c873d6d39ab1051a408141853f9978a"
server_key_iv: "3b05082d56ca95d3a765a7531f3d0951"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.450157 25313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "d84a54a8f8ee4879bf7c3463fd17adf2"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "9c873d6d39ab1051a408141853f9978a"
server_key_iv: "3b05082d56ca95d3a765a7531f3d0951"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.451447 25313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:43.452138 25328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:43.452289 25313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:43.452370 25313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "d84a54a8f8ee4879bf7c3463fd17adf2"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "9c873d6d39ab1051a408141853f9978a"
server_key_iv: "3b05082d56ca95d3a765a7531f3d0951"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.452455 25313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:43.470674 25313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:43.472469 25313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:43.472606 25313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:43.472823 25313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:43.473120 25313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:43.473152 25313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:43.473182 25313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:43.473199 25313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:43.478909 25313 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:33083
I20260928 12:34:43.478951 25441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:33083 every 8 connection(s)
I20260928 12:34:43.479288 25313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:34:43 dist-test-slave-g31s krb5kdc[25206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598883, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:43.487947 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 25313
I20260928 12:34:43.488071 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:34:43.488291 8244 external_mini_cluster.cc:1428] Setting key b6ad174713813a7b8e223e3279d3bda0
I20260928 12:34:43.488448 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.480049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:50519 (local address 127.8.13.62:42601)
0928 12:34:43.480118 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:43.480120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:43.480335 (+ 215us) server_negotiation.cc:407] Connection header received
0928 12:34:43.480816 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:43.480817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:43.480830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:43.480891 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:43.481897 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.482396 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.483379 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.483589 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.483637 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.485865 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:43.485866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:43.485866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:43.485874 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:43.487125 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:43.487512 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:43.487513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:43.487513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:43.487563 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:43.487740 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:43.487741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:43.487741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:43.487848 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:43.487892 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:43.488271 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 12:34:43.488357 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:34:43.488684 25444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.480021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:42601 (local address 127.8.13.1:50519)
0928 12:34:43.480209 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:43.480255 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:43.480741 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:43.480946 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:43.480956 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:43.481528 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:43.481724 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.481729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.482477 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.482478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.483309 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.483311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.483647 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.483647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.483752 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.484272 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:43.484281 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:43.485737 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:43.487199 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:43.487207 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:43.487210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:43.487450 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:43.487605 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:43.487606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:43.487606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:43.487701 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:43.487903 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:43.487912 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:43.488124 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 12:34:43.488249 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 12:34:43.488436 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260928 12:34:43.489185 25442 heartbeater.cc:347] Connected to a master server at 127.8.13.62:42601
I20260928 12:34:43.489305 25442 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:43.489508 25442 heartbeater.cc:511] Master 127.8.13.62:42601 requested a full tablet report, sending...
I20260928 12:34:43.490089 25251 ts_manager.cc:194] Registered new tserver with Master: d84a54a8f8ee4879bf7c3463fd17adf2 (127.8.13.1:33083)
I20260928 12:34:43.490722 25251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:50519
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:34:43.511528 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42601
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:37821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:43.594864 25447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:43.595093 25447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:43.595127 25447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:43.596336 25447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:43.596385 25447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:43.596422 25447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:34:43.598017 25447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:37821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:43.598272 25447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:43.598480 25447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:43.603262 25453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:43.603262 25452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:43.603262 25455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:43.603482 25447 server_base.cc:1034] running on GCE node
I20260928 12:34:43.603709 25447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:43.603917 25447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:43.605055 25447 hybrid_clock.cc:648] HybridClock initialized: now 1790598883605033 us; error 36 us; skew 500 ppm
Sep 28 12:34:43 dist-test-slave-g31s krb5kdc[25206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598883, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:43.607452 25447 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:34:43.607863 25447 webserver.cc:460] Webserver started at http://127.8.13.2:41921/ using document root <none> and password file <none>
I20260928 12:34:43.608065 25447 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:43.608112 25447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:43.608230 25447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:43.609248 25447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "d12e35d3a9c54d5097304359eac985dc"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "fae8a06a83ec00644e51edefe0ae2de5"
server_key_iv: "8b629dba540215c29d26276f6aa1a2ad"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.609570 25447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "d12e35d3a9c54d5097304359eac985dc"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "fae8a06a83ec00644e51edefe0ae2de5"
server_key_iv: "8b629dba540215c29d26276f6aa1a2ad"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.610790 25447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:34:43.611433 25462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:43.611577 25447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:34:43.611655 25447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "d12e35d3a9c54d5097304359eac985dc"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "fae8a06a83ec00644e51edefe0ae2de5"
server_key_iv: "8b629dba540215c29d26276f6aa1a2ad"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.611720 25447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:43.629578 25447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:43.631400 25447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:43.631529 25447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:43.631732 25447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:43.632021 25447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:43.632057 25447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:43.632086 25447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:43.632110 25447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:43.637718 25447 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:39463
I20260928 12:34:43.637778 25575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:39463 every 8 connection(s)
I20260928 12:34:43.638137 25447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:34:43 dist-test-slave-g31s krb5kdc[25206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598883, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:43.646242 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 25447
I20260928 12:34:43.646343 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:34:43.646534 8244 external_mini_cluster.cc:1428] Setting key d0c28a40a9c62a4e647bc7c5ca8407cf
I20260928 12:34:43.647678 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.638976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:49165 (local address 127.8.13.62:42601)
0928 12:34:43.639039 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:43.639041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:43.639197 (+ 156us) server_negotiation.cc:407] Connection header received
0928 12:34:43.639697 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:43.639699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:43.639711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:43.639776 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:43.640812 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.641323 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.642420 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.642627 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.642690 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.644871 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:43.644873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:43.644873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:43.644880 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:43.646170 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:43.646617 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:43.646618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:43.646618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:43.646671 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:43.646978 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:43.646978 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:43.646978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:43.647128 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:43.647193 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:43.647507 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 12:34:43.647594 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:34:43.647804 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.638909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:42601 (local address 127.8.13.2:49165)
0928 12:34:43.639072 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:43.639115 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:43.639597 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:43.639848 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:43.639852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:43.640465 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:43.640681 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.640686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.641423 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.641424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.642328 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.642332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.642704 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.642705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.642801 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.643307 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:43.643334 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:43.644767 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:43.646251 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:43.646261 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:43.646263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:43.646537 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:43.646737 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:43.646738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:43.646738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:43.646905 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:43.647196 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:43.647202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:43.647377 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 12:34:43.647469 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 12:34:43.647610 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260928 12:34:43.648389 25576 heartbeater.cc:347] Connected to a master server at 127.8.13.62:42601
I20260928 12:34:43.648519 25576 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:43.648716 25576 heartbeater.cc:511] Master 127.8.13.62:42601 requested a full tablet report, sending...
I20260928 12:34:43.649175 25251 ts_manager.cc:194] Registered new tserver with Master: d12e35d3a9c54d5097304359eac985dc (127.8.13.2:39463)
I20260928 12:34:43.649576 25251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:49165
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:34:43.673197 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42601
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:37821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:43.752441 25581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:43.752666 25581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:43.752700 25581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:43.753917 25581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:43.753966 25581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:43.754004 25581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:34:43.755640 25581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:37821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:42601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:43.755913 25581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:43.756153 25581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:43.761034 25589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:43.761041 25586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:43.761057 25587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:43.761209 25581 server_base.cc:1034] running on GCE node
I20260928 12:34:43.761520 25581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:43.761785 25581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:43.762962 25581 hybrid_clock.cc:648] HybridClock initialized: now 1790598883762948 us; error 30 us; skew 500 ppm
Sep 28 12:34:43 dist-test-slave-g31s krb5kdc[25206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598883, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:43.765328 25581 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:34:43.765749 25581 webserver.cc:460] Webserver started at http://127.8.13.3:35641/ using document root <none> and password file <none>
I20260928 12:34:43.766003 25581 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:43.766053 25581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:43.766175 25581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:43.767194 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "77c9a62f3a5443ba834d5493ce0c361f"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "4ef0aa9ba78c796791168075ba6ca1ef"
server_key_iv: "60ad4148c2026e16bf658fa22b7455b0"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.767524 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "77c9a62f3a5443ba834d5493ce0c361f"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "4ef0aa9ba78c796791168075ba6ca1ef"
server_key_iv: "60ad4148c2026e16bf658fa22b7455b0"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.768796 25581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:34:43.769579 25596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:43.769731 25581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:34:43.769845 25581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "77c9a62f3a5443ba834d5493ce0c361f"
format_stamp: "Formatted at 2026-09-28 12:34:43 on dist-test-slave-g31s"
server_key: "4ef0aa9ba78c796791168075ba6ca1ef"
server_key_iv: "60ad4148c2026e16bf658fa22b7455b0"
server_key_version: "encryptionkey@0"
I20260928 12:34:43.769912 25581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:43.783494 25581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:43.785703 25581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:43.785880 25581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:43.786099 25581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:43.786413 25581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:43.786448 25581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:43.786487 25581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:43.786511 25581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:43.792207 25581 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:33285
I20260928 12:34:43.792280 25709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:33285 every 8 connection(s)
I20260928 12:34:43.792608 25581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:34:43.797866 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 25581
I20260928 12:34:43.797969 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:34:43.798151 8244 external_mini_cluster.cc:1428] Setting key 64da80b18da6534dbb3caa5f90468bc5
Sep 28 12:34:43 dist-test-slave-g31s krb5kdc[25206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598883, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:43.802071 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.793325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:44395 (local address 127.8.13.62:42601)
0928 12:34:43.793386 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:43.793389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:43.793595 (+ 206us) server_negotiation.cc:407] Connection header received
0928 12:34:43.794151 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:43.794153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:43.794163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:43.794213 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:43.795194 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.795740 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.796773 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.796994 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.797052 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.799469 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:43.799471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:43.799471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:43.799479 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:43.800738 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:43.801126 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:43.801128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:43.801128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:43.801183 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:43.801409 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:43.801410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:43.801410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:43.801527 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:43.801581 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:43.801894 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 12:34:43.801976 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:34:43.802258 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.793306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:42601 (local address 127.8.13.3:44395)
0928 12:34:43.793474 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:43.793513 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:43.794058 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:43.794271 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:43.794274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:43.794866 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:43.795083 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.795087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.795840 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.795840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.796678 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.796681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.797065 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.797066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.797174 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.797671 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:43.797680 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:43.799377 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:43.800823 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:43.800831 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:43.800834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:43.801059 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:43.801239 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:43.801240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:43.801240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:43.801355 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:43.801579 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:43.801586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:43.801734 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 12:34:43.801870 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 12:34:43.802043 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260928 12:34:43.802687 25710 heartbeater.cc:347] Connected to a master server at 127.8.13.62:42601
I20260928 12:34:43.802786 25710 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:43.802986 25710 heartbeater.cc:511] Master 127.8.13.62:42601 requested a full tablet report, sending...
I20260928 12:34:43.803316 25251 ts_manager.cc:194] Registered new tserver with Master: 77c9a62f3a5443ba834d5493ce0c361f (127.8.13.3:33285)
I20260928 12:34:43.803676 25251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:44395
I20260928 12:34:43.809872 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:34:43.817123 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.811079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.8.13.62:42601)
0928 12:34:43.811126 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:43.811128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:43.811253 (+ 125us) server_negotiation.cc:407] Connection header received
0928 12:34:43.811290 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:43.811291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:43.811300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:43.811338 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:43.811848 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.812346 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.813176 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.813356 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.813403 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.814634 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:43.814635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:43.814635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:43.814642 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:43.815979 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:43.816334 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:43.816335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:43.816336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:43.816392 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:43.816564 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:43.816565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:43.816566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:43.816743 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:43.816816 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:43.816992 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 12:34:43.817054 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:34:43.818608 25251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:34:43.820199 25251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:34:43.826435 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.822276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:39463 (local address 127.0.0.1:59076)
0928 12:34:43.822937 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:43.822945 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:43.822984 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:43.823339 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:43.823340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:43.823342 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:43.823377 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.823378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.824460 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.824461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.825340 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.825342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.826147 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.826148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.826224 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.826239 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 12:34:43.826301 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:34:43.826353 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":50,"threads_started":1}
I20260928 12:34:43.826540 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.822469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:33285 (local address 127.0.0.1:60522)
0928 12:34:43.823044 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:43.823052 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:43.823094 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:43.823315 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:43.823317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:43.823320 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:43.823358 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.823360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.824267 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.824268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.825544 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.825547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.826284 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.826285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.826344 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.826348 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:34:43.826397 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:34:43.826444 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":48,"thread_start_us":48,"threads_started":1}
I20260928 12:34:43.826774 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.822388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59076 (local address 127.8.13.2:39463)
0928 12:34:43.823199 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:43.823203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:43.823211 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:34:43.823246 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:43.823248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:43.823262 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:43.823331 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:43.823427 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.824380 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.825410 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.826055 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.826119 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.826174 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:43.826654 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 12:34:43.826699 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":99,"threads_started":1}
I20260928 12:34:43.826941 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.822274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:33083 (local address 127.0.0.1:48778)
0928 12:34:43.822575 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:43.822614 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:43.822674 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:43.822875 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:43.822877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:43.822879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:43.822933 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.822934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.824215 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.824216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.825577 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.825579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.826700 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.826701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:43.826775 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.826780 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:34:43.826836 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:34:43.826878 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":35,"thread_start_us":68,"threads_started":1}
I20260928 12:34:43.827014 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.822398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.8.13.1:33083)
0928 12:34:43.822708 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:43.822712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:43.822721 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:34:43.822751 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:43.822752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:43.822764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:43.822866 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:43.823014 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.824117 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.825646 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.826345 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.826412 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.826461 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:43.826890 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 12:34:43.826933 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":69,"threads_started":1}
I20260928 12:34:43.827023 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.822561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.8.13.3:33285)
0928 12:34:43.822721 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:43.822725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:43.823096 (+ 371us) server_negotiation.cc:407] Connection header received
0928 12:34:43.823149 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:43.823151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:43.823166 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:43.823236 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:43.823430 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.824166 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.825638 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.826205 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.826273 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:43.826327 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:43.826420 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 12:34:43.826465 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260928 12:34:43.827391 25510 tablet_service.cc:1467] Processing CreateTablet for tablet 4d7f66d3cca34db492991cfb2bfa4998 (DEFAULT_TABLE table=test-table [id=b17e9019a8c64bcd988fa4d508d552e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:43.827714 25510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d7f66d3cca34db492991cfb2bfa4998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:43.827692 25644 tablet_service.cc:1467] Processing CreateTablet for tablet 4d7f66d3cca34db492991cfb2bfa4998 (DEFAULT_TABLE table=test-table [id=b17e9019a8c64bcd988fa4d508d552e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:43.827802 25376 tablet_service.cc:1467] Processing CreateTablet for tablet 4d7f66d3cca34db492991cfb2bfa4998 (DEFAULT_TABLE table=test-table [id=b17e9019a8c64bcd988fa4d508d552e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:43.827996 25644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d7f66d3cca34db492991cfb2bfa4998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:43.828068 25376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d7f66d3cca34db492991cfb2bfa4998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:43.830155 25724 tablet_bootstrap.cc:492] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: Bootstrap starting.
I20260928 12:34:43.830729 25724 tablet_bootstrap.cc:654] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:43.831115 25726 tablet_bootstrap.cc:492] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2: Bootstrap starting.
I20260928 12:34:43.831236 25724 log.cc:824] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:43.831703 25726 tablet_bootstrap.cc:654] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:43.832044 25724 tablet_bootstrap.cc:492] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: No bootstrap required, opened a new log
I20260928 12:34:43.832121 25724 ts_tablet_manager.cc:1397] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:34:43.832252 25726 log.cc:824] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:43.832594 25725 tablet_bootstrap.cc:492] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f: Bootstrap starting.
I20260928 12:34:43.832909 25726 tablet_bootstrap.cc:492] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2: No bootstrap required, opened a new log
I20260928 12:34:43.832969 25726 ts_tablet_manager.cc:1397] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:34:43.833206 25725 tablet_bootstrap.cc:654] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:43.833756 25725 log.cc:824] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:43.834156 25724 raft_consensus.cc:348] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33083 } } peers { permanent_uuid: "d12e35d3a9c54d5097304359eac985dc" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 39463 } } peers { permanent_uuid: "77c9a62f3a5443ba834d5493ce0c361f" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33285 } }
I20260928 12:34:43.834369 25724 raft_consensus.cc:374] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:43.834429 25724 raft_consensus.cc:729] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12e35d3a9c54d5097304359eac985dc, State: Initialized, Role: FOLLOWER
I20260928 12:34:43.834580 25724 consensus_queue.cc:260] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33083 } } peers { permanent_uuid: "d12e35d3a9c54d5097304359eac985dc" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 39463 } } peers { permanent_uuid: "77c9a62f3a5443ba834d5493ce0c361f" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33285 } }
I20260928 12:34:43.834659 25725 tablet_bootstrap.cc:492] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f: No bootstrap required, opened a new log
I20260928 12:34:43.834714 25725 ts_tablet_manager.cc:1397] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:34:43.834672 25726 raft_consensus.cc:348] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33083 } } peers { permanent_uuid: "d12e35d3a9c54d5097304359eac985dc" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 39463 } } peers { permanent_uuid: "77c9a62f3a5443ba834d5493ce0c361f" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33285 } }
I20260928 12:34:43.834867 25726 raft_consensus.cc:374] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:43.834909 25724 ts_tablet_manager.cc:1428] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:34:43.834935 25726 raft_consensus.cc:729] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84a54a8f8ee4879bf7c3463fd17adf2, State: Initialized, Role: FOLLOWER
I20260928 12:34:43.834973 25576 heartbeater.cc:503] Master 127.8.13.62:42601 was elected leader, sending a full tablet report...
I20260928 12:34:43.835042 25726 consensus_queue.cc:260] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33083 } } peers { permanent_uuid: "d12e35d3a9c54d5097304359eac985dc" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 39463 } } peers { permanent_uuid: "77c9a62f3a5443ba834d5493ce0c361f" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33285 } }
I20260928 12:34:43.835309 25726 ts_tablet_manager.cc:1428] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:34:43.835336 25442 heartbeater.cc:503] Master 127.8.13.62:42601 was elected leader, sending a full tablet report...
I20260928 12:34:43.836549 25725 raft_consensus.cc:348] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33083 } } peers { permanent_uuid: "d12e35d3a9c54d5097304359eac985dc" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 39463 } } peers { permanent_uuid: "77c9a62f3a5443ba834d5493ce0c361f" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33285 } }
I20260928 12:34:43.836681 25725 raft_consensus.cc:374] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:43.836700 25725 raft_consensus.cc:729] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c9a62f3a5443ba834d5493ce0c361f, State: Initialized, Role: FOLLOWER
I20260928 12:34:43.836777 25725 consensus_queue.cc:260] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33083 } } peers { permanent_uuid: "d12e35d3a9c54d5097304359eac985dc" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 39463 } } peers { permanent_uuid: "77c9a62f3a5443ba834d5493ce0c361f" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33285 } }
I20260928 12:34:43.837007 25725 ts_tablet_manager.cc:1428] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:34:43.837072 25710 heartbeater.cc:503] Master 127.8.13.62:42601 was elected leader, sending a full tablet report...
W20260928 12:34:43.888782 25577 tablet.cc:2369] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:34:43.980116 25443 tablet.cc:2369] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:34:43.996879 25732 raft_consensus.cc:482] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:34:43.997062 25732 raft_consensus.cc:504] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33083 } } peers { permanent_uuid: "d12e35d3a9c54d5097304359eac985dc" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 39463 } } peers { permanent_uuid: "77c9a62f3a5443ba834d5493ce0c361f" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33285 } }
I20260928 12:34:43.997424 25732 leader_election.cc:288] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d84a54a8f8ee4879bf7c3463fd17adf2 (127.8.13.1:33083), d12e35d3a9c54d5097304359eac985dc (127.8.13.2:39463)
I20260928 12:34:44.001308 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.997688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:39299 (local address 127.8.13.2:39463)
0928 12:34:43.997754 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:43.997756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:43.997992 (+ 236us) server_negotiation.cc:407] Connection header received
0928 12:34:43.998020 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:43.998021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:43.998031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:43.998080 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:43.998194 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.998880 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.000345 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.000929 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.001013 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.001023 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:44.001187 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 12:34:44.001230 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:34:44.001308 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.997736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:33079 (local address 127.8.13.1:33083)
0928 12:34:43.997777 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:43.997778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:43.997786 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:34:43.997837 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:43.997839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:43.997850 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:43.997920 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:43.998099 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.998871 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.000349 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.000929 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.001007 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.001021 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:44.001187 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 12:34:44.001230 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:34:44.001324 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.997639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:33083 (local address 127.8.13.3:33079)
0928 12:34:43.997688 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:43.997700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:43.997744 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:43.997946 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:43.997947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:43.997950 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:43.998026 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.998028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.998959 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.998959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.000256 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.000260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.001024 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.001025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.001118 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.001120 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:34:44.001176 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:34:44.001224 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":5}
I20260928 12:34:44.001331 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:43.997639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:39463 (local address 127.8.13.3:39299)
0928 12:34:43.997881 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:43.997888 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:43.997967 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:43.998092 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:43.998092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:43.998094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:43.998144 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:43.998145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:43.998958 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:43.998959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.000256 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.000260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.001030 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.001031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.001115 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.001120 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:34:44.001176 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:34:44.001224 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":162,"thread_start_us":126,"threads_started":1}
I20260928 12:34:44.001713 25396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d7f66d3cca34db492991cfb2bfa4998" candidate_uuid: "77c9a62f3a5443ba834d5493ce0c361f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" is_pre_election: true
I20260928 12:34:44.001909 25396 raft_consensus.cc:2454] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c9a62f3a5443ba834d5493ce0c361f in term 0.
I20260928 12:34:44.001906 25530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d7f66d3cca34db492991cfb2bfa4998" candidate_uuid: "77c9a62f3a5443ba834d5493ce0c361f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12e35d3a9c54d5097304359eac985dc" is_pre_election: true
I20260928 12:34:44.002056 25530 raft_consensus.cc:2454] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c9a62f3a5443ba834d5493ce0c361f in term 0.
I20260928 12:34:44.002146 25597 leader_election.cc:302] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c9a62f3a5443ba834d5493ce0c361f, d84a54a8f8ee4879bf7c3463fd17adf2; no voters:
I20260928 12:34:44.002274 25732 raft_consensus.cc:2781] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:34:44.002321 25732 raft_consensus.cc:482] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:34:44.002336 25732 raft_consensus.cc:3037] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:44.003026 25732 raft_consensus.cc:504] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33083 } } peers { permanent_uuid: "d12e35d3a9c54d5097304359eac985dc" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 39463 } } peers { permanent_uuid: "77c9a62f3a5443ba834d5493ce0c361f" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33285 } }
I20260928 12:34:44.003149 25732 leader_election.cc:288] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [CANDIDATE]: Term 1 election: Requested vote from peers d84a54a8f8ee4879bf7c3463fd17adf2 (127.8.13.1:33083), d12e35d3a9c54d5097304359eac985dc (127.8.13.2:39463)
I20260928 12:34:44.003289 25396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d7f66d3cca34db492991cfb2bfa4998" candidate_uuid: "77c9a62f3a5443ba834d5493ce0c361f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2"
I20260928 12:34:44.003304 25530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d7f66d3cca34db492991cfb2bfa4998" candidate_uuid: "77c9a62f3a5443ba834d5493ce0c361f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12e35d3a9c54d5097304359eac985dc"
I20260928 12:34:44.003364 25530 raft_consensus.cc:3037] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:44.003364 25396 raft_consensus.cc:3037] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:44.004042 25396 raft_consensus.cc:2454] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c9a62f3a5443ba834d5493ce0c361f in term 1.
I20260928 12:34:44.004060 25530 raft_consensus.cc:2454] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c9a62f3a5443ba834d5493ce0c361f in term 1.
I20260928 12:34:44.004206 25597 leader_election.cc:302] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c9a62f3a5443ba834d5493ce0c361f, d84a54a8f8ee4879bf7c3463fd17adf2; no voters:
I20260928 12:34:44.004310 25732 raft_consensus.cc:2781] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:44.004473 25732 raft_consensus.cc:686] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 1 LEADER]: Becoming Leader. State: Replica: 77c9a62f3a5443ba834d5493ce0c361f, State: Running, Role: LEADER
I20260928 12:34:44.004554 25732 consensus_queue.cc:237] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33083 } } peers { permanent_uuid: "d12e35d3a9c54d5097304359eac985dc" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 39463 } } peers { permanent_uuid: "77c9a62f3a5443ba834d5493ce0c361f" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33285 } }
I20260928 12:34:44.005353 25250 catalog_manager.cc:5495] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f reported cstate change: term changed from 0 to 1, leader changed from <none> to 77c9a62f3a5443ba834d5493ce0c361f (127.8.13.3). New cstate: current_term: 1 leader_uuid: "77c9a62f3a5443ba834d5493ce0c361f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12e35d3a9c54d5097304359eac985dc" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 39463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77c9a62f3a5443ba834d5493ce0c361f" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260928 12:34:44.011093 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.009027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.8.13.3:33285)
0928 12:34:44.009062 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:44.009063 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:44.009071 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:34:44.009115 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:44.009116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:44.009131 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:44.009185 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:34:44.009288 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.009732 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.010487 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.010668 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.010711 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.010807 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:34:44.010891 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:34:44.010945 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:44.011027 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 12:34:44.011043 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:34:44.012744 25530 raft_consensus.cc:1261] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc [term 1 FOLLOWER]: Refusing update from remote peer 77c9a62f3a5443ba834d5493ce0c361f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:44.012784 25396 raft_consensus.cc:1261] T 4d7f66d3cca34db492991cfb2bfa4998 P d84a54a8f8ee4879bf7c3463fd17adf2 [term 1 FOLLOWER]: Refusing update from remote peer 77c9a62f3a5443ba834d5493ce0c361f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:44.013090 25734 consensus_queue.cc:1035] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12e35d3a9c54d5097304359eac985dc" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 39463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:44.013212 25732 consensus_queue.cc:1035] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84a54a8f8ee4879bf7c3463fd17adf2" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:44.015827 25738 mvcc.cc:205] Tried to move back new op lower bound from 7334293028913811456 to 7334293028883275776. Current Snapshot: MvccSnapshot[applied={T|T < 7334293028913811456 or (T in {7334293028913811456})}]
I20260928 12:34:44.017571 25251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:44.017659 25251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:44.018363 25251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69bc4d2ad40e4d93983d4160154f39da: Sending DeleteTablet for 3 replicas of tablet 4d7f66d3cca34db492991cfb2bfa4998
I20260928 12:34:44.018672 25510 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d7f66d3cca34db492991cfb2bfa4998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:44 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:59076
I20260928 12:34:44.018666 25376 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d7f66d3cca34db492991cfb2bfa4998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:44 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:48778
I20260928 12:34:44.018903 25747 tablet_replica.cc:330] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: stopping tablet replica
I20260928 12:34:44.018997 25644 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d7f66d3cca34db492991cfb2bfa4998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:44 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:60522
I20260928 12:34:44.019042 25747 raft_consensus.cc:2229] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:34:44.019141 25747 raft_consensus.cc:2258] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:44.019332 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 25313
I20260928 12:34:44.019608 25747 ts_tablet_manager.cc:1905] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:44.021474 25747 ts_tablet_manager.cc:1918] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:44.021615 25747 log.cc:1196] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/4d7f66d3cca34db492991cfb2bfa4998
I20260928 12:34:44.021965 25747 ts_tablet_manager.cc:1939] T 4d7f66d3cca34db492991cfb2bfa4998 P d12e35d3a9c54d5097304359eac985dc: Deleting consensus metadata
I20260928 12:34:44.022431 25238 catalog_manager.cc:4841] TS d12e35d3a9c54d5097304359eac985dc (127.8.13.2:39463): tablet 4d7f66d3cca34db492991cfb2bfa4998 (table test-table [id=b17e9019a8c64bcd988fa4d508d552e5]) successfully deleted
W20260928 12:34:44.024941 25235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:33083 (error 108)
I20260928 12:34:44.025427 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.025297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:33083 (local address 127.0.0.1:48794)
0928 12:34:44.025352 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:44.025394 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 12:34:44.025466 25749 tablet_replica.cc:330] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f: stopping tablet replica
I20260928 12:34:44.025467 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 25447
I20260928 12:34:44.025565 25749 raft_consensus.cc:2229] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:34:44.025682 25749 raft_consensus.cc:2258] T 4d7f66d3cca34db492991cfb2bfa4998 P 77c9a62f3a5443ba834d5493ce0c361f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 12:34:44.027706 25235 catalog_manager.cc:4584] TS d84a54a8f8ee4879bf7c3463fd17adf2 (127.8.13.1:33083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d7f66d3cca34db492991cfb2bfa4998: Network error: Client connection negotiation failed: client connection to 127.8.13.1:33083: connect: Connection refused (error 111)
I20260928 12:34:44.030568 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 25581
I20260928 12:34:44.034641 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 25219
I20260928 12:34:44.034714 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.034589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:33285 (local address 127.0.0.1:60534)
0928 12:34:44.034650 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:44.034676 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.3:33285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
2026-09-28T12:34:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1929 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T12:34:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:44Z Disabled control of system clock
I20260928 12:34:44.061597 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45443
--webserver_interface=127.8.13.62
--webserver_port=0
--builtin_ntp_servers=127.8.13.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:45443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:34:44.142047 25754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:44.142293 25754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:44.142323 25754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:44.143584 25754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:44.143640 25754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:44.143657 25754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:44.143674 25754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:44.143690 25754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:44.145172 25754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:45443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:45443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:44.145452 25754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:44.145735 25754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:44.150617 25760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:44.150660 25759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:44.150662 25754 server_base.cc:1034] running on GCE node
W20260928 12:34:44.150617 25762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:44.151077 25754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:44.151399 25754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:44.152558 25754 hybrid_clock.cc:648] HybridClock initialized: now 1790598884152543 us; error 35 us; skew 500 ppm
I20260928 12:34:44.153985 25754 webserver.cc:460] Webserver started at http://127.8.13.62:33347/ using document root <none> and password file <none>
I20260928 12:34:44.154222 25754 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:44.154273 25754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:44.154405 25754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:44.155493 25754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "6fc237a1c387414c8dc87c1c3cc76be1"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "3181462d077047d7da691ec44a808ecd"
server_key_iv: "d0f5bbb5acf55145eed2af7d5e44127b"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.155833 25754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "6fc237a1c387414c8dc87c1c3cc76be1"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "3181462d077047d7da691ec44a808ecd"
server_key_iv: "d0f5bbb5acf55145eed2af7d5e44127b"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.157114 25754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:34:44.157779 25768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:44.157956 25754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:44.158043 25754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "6fc237a1c387414c8dc87c1c3cc76be1"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "3181462d077047d7da691ec44a808ecd"
server_key_iv: "d0f5bbb5acf55145eed2af7d5e44127b"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.158116 25754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:44.183439 25754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:44.183830 25754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:44.184011 25754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:44.188688 25754 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:45443
I20260928 12:34:44.188724 25820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:45443 every 8 connection(s)
I20260928 12:34:44.189098 25754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:44.190133 25821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:44.192576 25821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1: Bootstrap starting.
I20260928 12:34:44.193230 25821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:44.193514 25821 log.cc:824] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:44.194303 25821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1: No bootstrap required, opened a new log
I20260928 12:34:44.196043 25821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc237a1c387414c8dc87c1c3cc76be1" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45443 } }
I20260928 12:34:44.196249 25821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:44.196287 25821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc237a1c387414c8dc87c1c3cc76be1, State: Initialized, Role: FOLLOWER
I20260928 12:34:44.196425 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 25754
I20260928 12:34:44.196458 25821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc237a1c387414c8dc87c1c3cc76be1" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45443 } }
I20260928 12:34:44.196530 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:44.196553 25821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:44.196592 25821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:44.196645 25821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:44.196720 8244 external_mini_cluster.cc:1428] Setting key 1bab6c072d5a6dfdf04334ee60aaa4e7
I20260928 12:34:44.197396 25821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc237a1c387414c8dc87c1c3cc76be1" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45443 } }
I20260928 12:34:44.197528 25821 leader_election.cc:302] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fc237a1c387414c8dc87c1c3cc76be1; no voters:
I20260928 12:34:44.197695 25821 leader_election.cc:288] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:44.197737 25826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:44.197927 25826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [term 1 LEADER]: Becoming Leader. State: Replica: 6fc237a1c387414c8dc87c1c3cc76be1, State: Running, Role: LEADER
I20260928 12:34:44.198006 25821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:44.198103 25826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc237a1c387414c8dc87c1c3cc76be1" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45443 } }
I20260928 12:34:44.198485 25826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc237a1c387414c8dc87c1c3cc76be1. Latest consensus state: current_term: 1 leader_uuid: "6fc237a1c387414c8dc87c1c3cc76be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc237a1c387414c8dc87c1c3cc76be1" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45443 } } }
I20260928 12:34:44.198568 25826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:44.198519 25827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fc237a1c387414c8dc87c1c3cc76be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc237a1c387414c8dc87c1c3cc76be1" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 45443 } } }
I20260928 12:34:44.198625 25827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:44.199542 25832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:44.200074 25832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:44.201596 25832 catalog_manager.cc:1347] Generated new cluster ID: a4a1e28d7ac84057a6aefe0caee77110
I20260928 12:34:44.201646 25832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:44.213006 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.197285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.8.13.62:45443)
0928 12:34:44.197520 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:44.197529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:44.197563 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:34:44.197690 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:44.197699 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:44.197783 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:44.198246 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:44.198361 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.201586 (+ 3225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.210660 (+ 9074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.211394 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.211545 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.212240 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:44.212245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:44.212245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:44.212470 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:44.212515 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:44.212531 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:44.212599 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:44.212661 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 12:34:44.212750 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260928 12:34:44.218705 25832 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:44.219424 25832 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:44.227240 25832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1: Generated new TSK 0
I20260928 12:34:44.227439 25832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:34:44.233443 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45443
--builtin_ntp_servers=127.8.13.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:34:44.314878 25845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:44.315097 25845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:44.315128 25845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:44.316264 25845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:44.316303 25845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:44.316337 25845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:34:44.317667 25845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:44.317929 25845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:44.318222 25845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:44.322954 25853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:44.323010 25850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:44.322971 25851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:44.323262 25845 server_base.cc:1034] running on GCE node
I20260928 12:34:44.323434 25845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:44.323679 25845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:44.324815 25845 hybrid_clock.cc:648] HybridClock initialized: now 1790598884324798 us; error 29 us; skew 500 ppm
I20260928 12:34:44.326069 25845 webserver.cc:460] Webserver started at http://127.8.13.1:38907/ using document root <none> and password file <none>
I20260928 12:34:44.326303 25845 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:44.326349 25845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:44.326473 25845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:44.327499 25845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "6f153837747245949f5ec12e88cbcd1f"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "303ed7dfc8790f5d1e394f8f50775c67"
server_key_iv: "689e3d1c04680251a707f86b15c3d089"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.327817 25845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "6f153837747245949f5ec12e88cbcd1f"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "303ed7dfc8790f5d1e394f8f50775c67"
server_key_iv: "689e3d1c04680251a707f86b15c3d089"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.329082 25845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:34:44.329698 25859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:44.329880 25845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:44.329977 25845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "6f153837747245949f5ec12e88cbcd1f"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "303ed7dfc8790f5d1e394f8f50775c67"
server_key_iv: "689e3d1c04680251a707f86b15c3d089"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.330037 25845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:44.344973 25845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:44.345280 25845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:44.345387 25845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:44.345597 25845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:44.345943 25845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:44.345988 25845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:44.346019 25845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:44.346035 25845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:44.351702 25845 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:34153
I20260928 12:34:44.351747 25972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:34153 every 8 connection(s)
I20260928 12:34:44.352087 25845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:34:44.357204 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.352806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:39839 (local address 127.8.13.62:45443)
0928 12:34:44.352899 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:44.352901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:44.353088 (+ 187us) server_negotiation.cc:407] Connection header received
0928 12:34:44.353594 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:44.353595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:44.353607 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:44.353670 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:44.354423 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.354970 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.355950 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.356159 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.356216 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.356961 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:44.356962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:44.356962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:44.356967 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:44.356982 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:44.356986 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:44.357028 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:44.357104 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 12:34:44.357136 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 12:34:44.357360 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.352749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45443 (local address 127.8.13.1:39839)
0928 12:34:44.352942 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:44.352986 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:44.353508 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:44.353731 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:44.353735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:44.353743 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:44.354307 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.354312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.355048 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.355049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.355863 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.355865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.356226 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.356227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.356336 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.356827 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:34:44.356827 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:44.356857 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:34:44.356881 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:44.357034 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:44.357036 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:44.357039 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:44.357088 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:34:44.357171 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260928 12:34:44.357860 25973 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45443
I20260928 12:34:44.357975 25973 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:44.358177 25973 heartbeater.cc:511] Master 127.8.13.62:45443 requested a full tablet report, sending...
I20260928 12:34:44.358589 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 25845
I20260928 12:34:44.358698 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:34:44.358789 25777 ts_manager.cc:194] Registered new tserver with Master: 6f153837747245949f5ec12e88cbcd1f (127.8.13.1:34153)
I20260928 12:34:44.358909 8244 external_mini_cluster.cc:1428] Setting key 1a14fdf5e2532577341365a57a5d764d
I20260928 12:34:44.359571 25777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.1:39839
I20260928 12:34:44.360081 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45443
--builtin_ntp_servers=127.8.13.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:34:44.441067 25976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:44.441293 25976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:44.441316 25976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:44.442523 25976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:44.442572 25976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:44.442598 25976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:34:44.443960 25976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:44.444173 25976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:44.444444 25976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:44.449361 25984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:44.449620 25982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:44.449636 25981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:44.449636 25976 server_base.cc:1034] running on GCE node
I20260928 12:34:44.449952 25976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:44.450208 25976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:44.451359 25976 hybrid_clock.cc:648] HybridClock initialized: now 1790598884451338 us; error 35 us; skew 500 ppm
I20260928 12:34:44.452682 25976 webserver.cc:460] Webserver started at http://127.8.13.2:46529/ using document root <none> and password file <none>
I20260928 12:34:44.452939 25976 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:44.453008 25976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:44.453145 25976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:44.454368 25976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "fa659d5461e845a4a1136bf13e1d120c"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "1199cfc79a15fce78b2c0a10fca04c5c"
server_key_iv: "0a4f55958a3733931700b9fc7409e4d2"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.454713 25976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "fa659d5461e845a4a1136bf13e1d120c"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "1199cfc79a15fce78b2c0a10fca04c5c"
server_key_iv: "0a4f55958a3733931700b9fc7409e4d2"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.455920 25976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:34:44.456676 25990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:44.456813 25976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:44.456894 25976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "fa659d5461e845a4a1136bf13e1d120c"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "1199cfc79a15fce78b2c0a10fca04c5c"
server_key_iv: "0a4f55958a3733931700b9fc7409e4d2"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.456972 25976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:44.465641 25976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:44.466037 25976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:44.466194 25976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:44.466416 25976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:44.466758 25976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:44.466796 25976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:44.466883 25976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:44.466895 25976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:44.472325 25976 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:43909
I20260928 12:34:44.472388 26103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:43909 every 8 connection(s)
I20260928 12:34:44.472703 25976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:34:44.474813 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 25976
I20260928 12:34:44.474918 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:34:44.475090 8244 external_mini_cluster.cc:1428] Setting key 3bb3e5edb03fd6cda106203ad68a6676
I20260928 12:34:44.476004 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45443
--builtin_ntp_servers=127.8.13.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 12:34:44.478050 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.473417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:46243 (local address 127.8.13.62:45443)
0928 12:34:44.473481 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:44.473483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:44.473695 (+ 212us) server_negotiation.cc:407] Connection header received
0928 12:34:44.474263 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:44.474264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:44.474276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:44.474327 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:44.475029 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.475606 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.476650 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.476849 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.476904 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.477757 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:44.477758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:44.477758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:44.477763 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:44.477779 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:44.477783 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:44.477871 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:44.477958 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 12:34:44.477986 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:34:44.478317 26106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.473404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45443 (local address 127.8.13.2:46243)
0928 12:34:44.473581 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:44.473616 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:44.474163 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:44.474401 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:44.474405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:44.474415 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:44.474920 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.474925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.475692 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.475693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.476568 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.476570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.476972 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.476973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.477090 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.477591 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:34:44.477592 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:44.477635 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:34:44.477667 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:44.477880 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:44.477884 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:44.477888 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:34:44.477957 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 12:34:44.478080 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260928 12:34:44.478863 26104 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45443
I20260928 12:34:44.478958 26104 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:44.479141 26104 heartbeater.cc:511] Master 127.8.13.62:45443 requested a full tablet report, sending...
I20260928 12:34:44.479512 25777 ts_manager.cc:194] Registered new tserver with Master: fa659d5461e845a4a1136bf13e1d120c (127.8.13.2:43909)
I20260928 12:34:44.479878 25777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.2:46243
W20260928 12:34:44.556522 26107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:44.556741 26107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:44.556771 26107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:44.558022 26107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:44.558077 26107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:44.558113 26107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:34:44.559607 26107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:45443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:44.559898 26107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:44.560230 26107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:44.565608 26113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:44.565596 26112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:44.565656 26107 server_base.cc:1034] running on GCE node
W20260928 12:34:44.565595 26115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:44.565990 26107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:44.566216 26107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:44.567386 26107 hybrid_clock.cc:648] HybridClock initialized: now 1790598884567371 us; error 29 us; skew 500 ppm
I20260928 12:34:44.568688 26107 webserver.cc:460] Webserver started at http://127.8.13.3:46689/ using document root <none> and password file <none>
I20260928 12:34:44.568940 26107 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:44.568991 26107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:44.569115 26107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:44.570196 26107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "1eff8d1569b9a3a0955b371dca7038b9"
server_key_iv: "cd7e98699559bb9775707e6d48e39bed"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.570542 26107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "1eff8d1569b9a3a0955b371dca7038b9"
server_key_iv: "cd7e98699559bb9775707e6d48e39bed"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.571776 26107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:34:44.572556 26121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:44.572722 26107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:34:44.572820 26107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a"
format_stamp: "Formatted at 2026-09-28 12:34:44 on dist-test-slave-g31s"
server_key: "1eff8d1569b9a3a0955b371dca7038b9"
server_key_iv: "cd7e98699559bb9775707e6d48e39bed"
server_key_version: "encryptionkey@0"
I20260928 12:34:44.572883 26107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:44.588053 26107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:44.588392 26107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:44.588529 26107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:44.588763 26107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:44.589128 26107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:44.589160 26107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:44.589191 26107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:44.589215 26107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:44.594815 26107 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:45551
I20260928 12:34:44.594888 26234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:45551 every 8 connection(s)
I20260928 12:34:44.595206 26107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:34:44.600786 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.595920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:58803 (local address 127.8.13.62:45443)
0928 12:34:44.596013 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:44.596015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:44.596212 (+ 197us) server_negotiation.cc:407] Connection header received
0928 12:34:44.596715 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:44.596717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:44.596727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:44.596768 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:44.597463 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.598059 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.599195 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.599393 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.599456 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.600450 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:44.600452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:44.600452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:44.600458 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:44.600484 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:44.600489 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:44.600549 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:44.600661 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 12:34:44.600696 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 12:34:44.600992 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.595897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:45443 (local address 127.8.13.3:58803)
0928 12:34:44.596090 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:44.596131 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:44.596615 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:44.596828 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:44.596831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:44.596840 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:44.597352 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.597357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.598175 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.598176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.599112 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.599117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.599495 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.599496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.599648 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.600295 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:34:44.600295 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:44.600327 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:34:44.600356 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:44.600558 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:44.600563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:44.600567 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:34:44.600649 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:34:44.600764 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260928 12:34:44.601294 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 26107
I20260928 12:34:44.601398 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:34:44.601529 26235 heartbeater.cc:347] Connected to a master server at 127.8.13.62:45443
I20260928 12:34:44.601608 8244 external_mini_cluster.cc:1428] Setting key 34d5a73f4393898abf711d37e05a1293
I20260928 12:34:44.601631 26235 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:44.601861 26235 heartbeater.cc:511] Master 127.8.13.62:45443 requested a full tablet report, sending...
I20260928 12:34:44.602257 25777 ts_manager.cc:194] Registered new tserver with Master: 06e4053ce52d40b0b17dc01b9cc2ad0a (127.8.13.3:45551)
I20260928 12:34:44.602562 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:34:44.602715 25777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.3:58803
I20260928 12:34:44.606590 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.603974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.8.13.62:45443)
0928 12:34:44.604024 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:44.604026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:44.604124 (+ 98us) server_negotiation.cc:407] Connection header received
0928 12:34:44.604181 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:44.604182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:44.604192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:44.604227 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:44.604366 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.604822 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.605513 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.605688 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.605734 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.606376 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:44.606378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:44.606378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:44.606382 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:44.606396 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:44.606400 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:44.606444 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:44.606481 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 12:34:44.606504 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:34:44.607998 25777 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:34:44.609540 25777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:34:44.615702 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.611845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.8.13.3:45551)
0928 12:34:44.612382 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:44.612385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:44.612395 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:34:44.612427 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:44.612429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:44.612441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:44.612644 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:44.612693 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.613394 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.614739 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.615340 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.615402 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.615502 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:44.615583 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 12:34:44.615617 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":104,"threads_started":1}
I20260928 12:34:44.615693 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.611700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:45551 (local address 127.0.0.1:48082)
0928 12:34:44.612178 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:44.612188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:44.612260 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:44.612530 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:44.612531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:44.612532 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:44.612580 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.612582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.613465 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.613466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.614654 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.614657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.615411 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.615412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.615488 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.615501 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:34:44.615560 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:34:44.615614 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260928 12:34:44.616067 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.611918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:34153 (local address 127.0.0.1:33934)
0928 12:34:44.612499 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:44.612508 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:44.612554 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:44.612706 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:44.612707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:44.612708 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:44.612740 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.612742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.613977 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.613978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.614909 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.614912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.615854 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.615855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.615916 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.615921 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:34:44.615962 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 12:34:44.615998 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":66,"threads_started":1}
I20260928 12:34:44.616295 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.612008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.8.13.1:34153)
0928 12:34:44.612189 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:44.612194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:44.612558 (+ 364us) server_negotiation.cc:407] Connection header received
0928 12:34:44.612611 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:44.612614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:44.612629 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:44.612702 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:44.612802 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.613904 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.615000 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.615687 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.615740 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.615817 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:44.616176 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 12:34:44.616217 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260928 12:34:44.616365 26169 tablet_service.cc:1467] Processing CreateTablet for tablet 4a85f973255d4481b866939decfcf056 (DEFAULT_TABLE table=test-table [id=96df841efca2426cba53c58e8e6201f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:44.616595 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.612103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.8.13.2:43909)
0928 12:34:44.612269 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:44.612273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:44.612283 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:34:44.612320 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:44.612322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:44.612338 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:44.612411 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:44.612617 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.613743 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.614930 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.615638 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.615706 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.616467 (+ 761us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:44.616489 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 12:34:44.616519 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260928 12:34:44.616747 26169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a85f973255d4481b866939decfcf056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:44.616762 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:44.611700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:43909 (local address 127.0.0.1:33512)
0928 12:34:44.612036 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:44.612074 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:44.612141 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:44.612458 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:44.612461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:44.612464 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:44.612533 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.612535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.613843 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.613844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.614862 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:44.614864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:44.616080 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:44.616081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:44.616273 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:44.616276 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:44.616321 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:34:44.616701 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":84,"threads_started":1}
I20260928 12:34:44.616969 25907 tablet_service.cc:1467] Processing CreateTablet for tablet 4a85f973255d4481b866939decfcf056 (DEFAULT_TABLE table=test-table [id=96df841efca2426cba53c58e8e6201f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:44.617226 25907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a85f973255d4481b866939decfcf056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:44.617386 26038 tablet_service.cc:1467] Processing CreateTablet for tablet 4a85f973255d4481b866939decfcf056 (DEFAULT_TABLE table=test-table [id=96df841efca2426cba53c58e8e6201f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:44.617686 26038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a85f973255d4481b866939decfcf056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:44.619625 26249 tablet_bootstrap.cc:492] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f: Bootstrap starting.
I20260928 12:34:44.619926 26250 tablet_bootstrap.cc:492] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: Bootstrap starting.
I20260928 12:34:44.620388 26249 tablet_bootstrap.cc:654] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:44.620483 26250 tablet_bootstrap.cc:654] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:44.620685 26251 tablet_bootstrap.cc:492] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: Bootstrap starting.
I20260928 12:34:44.620762 26250 log.cc:824] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:44.620841 26249 log.cc:824] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:44.621276 26251 tablet_bootstrap.cc:654] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:44.621471 26250 tablet_bootstrap.cc:492] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: No bootstrap required, opened a new log
I20260928 12:34:44.621479 26249 tablet_bootstrap.cc:492] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f: No bootstrap required, opened a new log
I20260928 12:34:44.621541 26249 ts_tablet_manager.cc:1397] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:34:44.621541 26250 ts_tablet_manager.cc:1397] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:34:44.621546 26251 log.cc:824] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:44.622159 26251 tablet_bootstrap.cc:492] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: No bootstrap required, opened a new log
I20260928 12:34:44.622216 26251 ts_tablet_manager.cc:1397] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:34:44.623970 26249 raft_consensus.cc:348] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa659d5461e845a4a1136bf13e1d120c" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 43909 } } peers { permanent_uuid: "6f153837747245949f5ec12e88cbcd1f" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 34153 } } peers { permanent_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 45551 } }
I20260928 12:34:44.623970 26250 raft_consensus.cc:348] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa659d5461e845a4a1136bf13e1d120c" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 43909 } } peers { permanent_uuid: "6f153837747245949f5ec12e88cbcd1f" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 34153 } } peers { permanent_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 45551 } }
I20260928 12:34:44.624137 26249 raft_consensus.cc:374] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:44.624137 26250 raft_consensus.cc:374] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:44.624171 26250 raft_consensus.cc:729] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e4053ce52d40b0b17dc01b9cc2ad0a, State: Initialized, Role: FOLLOWER
I20260928 12:34:44.624171 26249 raft_consensus.cc:729] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f153837747245949f5ec12e88cbcd1f, State: Initialized, Role: FOLLOWER
I20260928 12:34:44.624167 26251 raft_consensus.cc:348] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa659d5461e845a4a1136bf13e1d120c" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 43909 } } peers { permanent_uuid: "6f153837747245949f5ec12e88cbcd1f" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 34153 } } peers { permanent_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 45551 } }
I20260928 12:34:44.624318 26251 raft_consensus.cc:374] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:44.624308 26250 consensus_queue.cc:260] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa659d5461e845a4a1136bf13e1d120c" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 43909 } } peers { permanent_uuid: "6f153837747245949f5ec12e88cbcd1f" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 34153 } } peers { permanent_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 45551 } }
I20260928 12:34:44.624308 26249 consensus_queue.cc:260] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa659d5461e845a4a1136bf13e1d120c" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 43909 } } peers { permanent_uuid: "6f153837747245949f5ec12e88cbcd1f" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 34153 } } peers { permanent_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 45551 } }
I20260928 12:34:44.624357 26251 raft_consensus.cc:729] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa659d5461e845a4a1136bf13e1d120c, State: Initialized, Role: FOLLOWER
I20260928 12:34:44.624481 26251 consensus_queue.cc:260] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa659d5461e845a4a1136bf13e1d120c" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 43909 } } peers { permanent_uuid: "6f153837747245949f5ec12e88cbcd1f" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 34153 } } peers { permanent_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 45551 } }
I20260928 12:34:44.624562 26250 ts_tablet_manager.cc:1428] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:34:44.624562 26249 ts_tablet_manager.cc:1428] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:34:44.624673 25973 heartbeater.cc:503] Master 127.8.13.62:45443 was elected leader, sending a full tablet report...
I20260928 12:34:44.624723 26251 ts_tablet_manager.cc:1428] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:34:44.624732 26235 heartbeater.cc:503] Master 127.8.13.62:45443 was elected leader, sending a full tablet report...
I20260928 12:34:44.624838 26104 heartbeater.cc:503] Master 127.8.13.62:45443 was elected leader, sending a full tablet report...
W20260928 12:34:44.723270 26105 tablet.cc:2369] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:34:44.845774 26236 tablet.cc:2369] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:34:44.852780 25974 tablet.cc:2369] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:34:45.019618 26257 raft_consensus.cc:482] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:34:45.019759 26257 raft_consensus.cc:504] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa659d5461e845a4a1136bf13e1d120c" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 43909 } } peers { permanent_uuid: "6f153837747245949f5ec12e88cbcd1f" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 34153 } } peers { permanent_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 45551 } }
I20260928 12:34:45.020123 26257 leader_election.cc:288] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f153837747245949f5ec12e88cbcd1f (127.8.13.1:34153), 06e4053ce52d40b0b17dc01b9cc2ad0a (127.8.13.3:45551)
I20260928 12:34:45.023998 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:45.020394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:52355 (local address 127.8.13.1:34153)
0928 12:34:45.020458 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:45.020460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:45.020513 (+ 53us) server_negotiation.cc:407] Connection header received
0928 12:34:45.020607 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:45.020607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:45.020619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:45.020673 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:45.020837 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:45.021595 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:45.023093 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:45.023643 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:45.023717 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:45.023727 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:45.023883 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 12:34:45.023922 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:34:45.023996 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:45.020426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:46167 (local address 127.8.13.3:45551)
0928 12:34:45.020467 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:45.020469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:45.020523 (+ 54us) server_negotiation.cc:407] Connection header received
0928 12:34:45.020607 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:45.020608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:45.020617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:45.020673 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:45.020865 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:45.021595 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:45.023093 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:45.023643 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:45.023717 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:45.023727 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:45.023883 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 12:34:45.023922 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:34:45.024066 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:45.020319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:34153 (local address 127.8.13.2:52355)
0928 12:34:45.020448 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:45.020459 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:45.020525 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:45.020687 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:45.020688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:45.020690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:45.020766 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:45.020768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:45.021676 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:45.021677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:45.022985 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:45.022986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:45.023725 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:45.023726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:45.023803 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:45.023811 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 12:34:45.023868 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:34:45.023920 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":33,"thread_start_us":53,"threads_started":1}
I20260928 12:34:45.024066 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:45.020319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:45551 (local address 127.8.13.2:46167)
0928 12:34:45.020447 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:45.020459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:45.020525 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:45.020687 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:45.020688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:45.020690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:45.020768 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:45.020769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:45.021676 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:45.021677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:45.022981 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:45.022985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:45.023726 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:45.023726 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:45.023804 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:45.023813 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 12:34:45.023868 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 12:34:45.023920 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":86,"thread_start_us":67,"threads_started":1}
I20260928 12:34:45.024488 25927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a85f973255d4481b866939decfcf056" candidate_uuid: "fa659d5461e845a4a1136bf13e1d120c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f153837747245949f5ec12e88cbcd1f" is_pre_election: true
I20260928 12:34:45.024560 26189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a85f973255d4481b866939decfcf056" candidate_uuid: "fa659d5461e845a4a1136bf13e1d120c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" is_pre_election: true
I20260928 12:34:45.024626 25927 raft_consensus.cc:2454] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa659d5461e845a4a1136bf13e1d120c in term 0.
I20260928 12:34:45.024721 26189 raft_consensus.cc:2454] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa659d5461e845a4a1136bf13e1d120c in term 0.
I20260928 12:34:45.024859 25991 leader_election.cc:302] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f153837747245949f5ec12e88cbcd1f, fa659d5461e845a4a1136bf13e1d120c; no voters:
I20260928 12:34:45.025028 26257 raft_consensus.cc:2781] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:34:45.025064 26257 raft_consensus.cc:482] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:34:45.025086 26257 raft_consensus.cc:3037] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:45.025775 26257 raft_consensus.cc:504] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa659d5461e845a4a1136bf13e1d120c" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 43909 } } peers { permanent_uuid: "6f153837747245949f5ec12e88cbcd1f" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 34153 } } peers { permanent_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 45551 } }
I20260928 12:34:45.025941 26257 leader_election.cc:288] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [CANDIDATE]: Term 1 election: Requested vote from peers 6f153837747245949f5ec12e88cbcd1f (127.8.13.1:34153), 06e4053ce52d40b0b17dc01b9cc2ad0a (127.8.13.3:45551)
I20260928 12:34:45.026078 25927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a85f973255d4481b866939decfcf056" candidate_uuid: "fa659d5461e845a4a1136bf13e1d120c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f153837747245949f5ec12e88cbcd1f"
I20260928 12:34:45.026099 26189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a85f973255d4481b866939decfcf056" candidate_uuid: "fa659d5461e845a4a1136bf13e1d120c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a"
I20260928 12:34:45.026160 25927 raft_consensus.cc:3037] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:45.026160 26189 raft_consensus.cc:3037] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:45.026984 25927 raft_consensus.cc:2454] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa659d5461e845a4a1136bf13e1d120c in term 1.
I20260928 12:34:45.026984 26189 raft_consensus.cc:2454] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa659d5461e845a4a1136bf13e1d120c in term 1.
I20260928 12:34:45.027160 25992 leader_election.cc:302] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e4053ce52d40b0b17dc01b9cc2ad0a, fa659d5461e845a4a1136bf13e1d120c; no voters:
I20260928 12:34:45.027266 26257 raft_consensus.cc:2781] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:45.027412 26257 raft_consensus.cc:686] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 1 LEADER]: Becoming Leader. State: Replica: fa659d5461e845a4a1136bf13e1d120c, State: Running, Role: LEADER
I20260928 12:34:45.027506 26257 consensus_queue.cc:237] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa659d5461e845a4a1136bf13e1d120c" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 43909 } } peers { permanent_uuid: "6f153837747245949f5ec12e88cbcd1f" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 34153 } } peers { permanent_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 45551 } }
I20260928 12:34:45.028275 25784 catalog_manager.cc:5495] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c reported cstate change: term changed from 0 to 1, leader changed from <none> to fa659d5461e845a4a1136bf13e1d120c (127.8.13.2). New cstate: current_term: 1 leader_uuid: "fa659d5461e845a4a1136bf13e1d120c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa659d5461e845a4a1136bf13e1d120c" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f153837747245949f5ec12e88cbcd1f" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 34153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 45551 } health_report { overall_health: UNKNOWN } } }
I20260928 12:34:45.057631 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:45.055510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.8.13.2:43909)
0928 12:34:45.055558 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:45.055560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:45.055569 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:34:45.055628 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:45.055629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:45.055646 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:45.055699 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:34:45.055788 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:45.056254 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:45.057020 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:45.057207 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:45.057258 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:45.057355 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:34:45.057459 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:34:45.057517 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:45.057530 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 12:34:45.057547 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:34:45.059541 26189 raft_consensus.cc:1261] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a [term 1 FOLLOWER]: Refusing update from remote peer fa659d5461e845a4a1136bf13e1d120c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:45.059551 25927 raft_consensus.cc:1261] T 4a85f973255d4481b866939decfcf056 P 6f153837747245949f5ec12e88cbcd1f [term 1 FOLLOWER]: Refusing update from remote peer fa659d5461e845a4a1136bf13e1d120c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:45.059939 26260 consensus_queue.cc:1035] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f153837747245949f5ec12e88cbcd1f" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 34153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:45.060101 26257 consensus_queue.cc:1035] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e4053ce52d40b0b17dc01b9cc2ad0a" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:45.062829 26263 mvcc.cc:205] Tried to move back new op lower bound from 7334293033201106944 to 7334293033073209344. Current Snapshot: MvccSnapshot[applied={T|T < 7334293033201106944}]
I20260928 12:34:45.063203 26265 mvcc.cc:205] Tried to move back new op lower bound from 7334293033201106944 to 7334293033073209344. Current Snapshot: MvccSnapshot[applied={T|T < 7334293033201106944}]
I20260928 12:34:45.063591 26264 mvcc.cc:205] Tried to move back new op lower bound from 7334293033201106944 to 7334293033073209344. Current Snapshot: MvccSnapshot[applied={T|T < 7334293033201106944 or (T in {7334293033201106944})}]
I20260928 12:34:45.065255 25777 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:45.065352 25777 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:45.066143 25777 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fc237a1c387414c8dc87c1c3cc76be1: Sending DeleteTablet for 3 replicas of tablet 4a85f973255d4481b866939decfcf056
I20260928 12:34:45.066463 26038 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a85f973255d4481b866939decfcf056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:45 UTC) from {username='slave'} at 127.0.0.1:33512
I20260928 12:34:45.066506 26169 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a85f973255d4481b866939decfcf056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:45 UTC) from {username='slave'} at 127.0.0.1:48082
I20260928 12:34:45.066548 25907 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a85f973255d4481b866939decfcf056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:45 UTC) from {username='slave'} at 127.0.0.1:33934
I20260928 12:34:45.066680 26273 tablet_replica.cc:330] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: stopping tablet replica
I20260928 12:34:45.066778 26273 raft_consensus.cc:2229] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:34:45.066969 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 25845
I20260928 12:34:45.066905 26273 raft_consensus.cc:2258] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:45.068886 26273 ts_tablet_manager.cc:1905] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:45.069291 26274 tablet_replica.cc:330] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: stopping tablet replica
I20260928 12:34:45.069418 26274 raft_consensus.cc:2229] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:34:45.069532 26274 raft_consensus.cc:2258] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:45.070056 26274 ts_tablet_manager.cc:1905] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:45.070773 26273 ts_tablet_manager.cc:1918] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:45.070850 26273 log.cc:1196] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/4a85f973255d4481b866939decfcf056
I20260928 12:34:45.071074 26273 ts_tablet_manager.cc:1939] T 4a85f973255d4481b866939decfcf056 P fa659d5461e845a4a1136bf13e1d120c: Deleting consensus metadata
I20260928 12:34:45.071506 25769 catalog_manager.cc:4841] TS fa659d5461e845a4a1136bf13e1d120c (127.8.13.2:43909): tablet 4a85f973255d4481b866939decfcf056 (table test-table [id=96df841efca2426cba53c58e8e6201f1]) successfully deleted
I20260928 12:34:45.072095 26274 ts_tablet_manager.cc:1918] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:45.072183 26274 log.cc:1196] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/4a85f973255d4481b866939decfcf056
I20260928 12:34:45.072474 26274 ts_tablet_manager.cc:1939] T 4a85f973255d4481b866939decfcf056 P 06e4053ce52d40b0b17dc01b9cc2ad0a: Deleting consensus metadata
I20260928 12:34:45.073040 25770 catalog_manager.cc:4841] TS 06e4053ce52d40b0b17dc01b9cc2ad0a (127.8.13.3:45551): tablet 4a85f973255d4481b866939decfcf056 (table test-table [id=96df841efca2426cba53c58e8e6201f1]) successfully deleted
W20260928 12:34:45.073418 25769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:34153 (error 108)
I20260928 12:34:45.073520 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 25976
I20260928 12:34:45.074982 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:45.074812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:34153 (local address 127.0.0.1:33940)
0928 12:34:45.074879 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:45.074934 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 12:34:45.075133 25769 catalog_manager.cc:4584] TS 6f153837747245949f5ec12e88cbcd1f (127.8.13.1:34153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a85f973255d4481b866939decfcf056: Network error: Client connection negotiation failed: client connection to 127.8.13.1:34153: connect: Connection refused (error 111)
I20260928 12:34:45.078157 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 26107
I20260928 12:34:45.083036 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 25754
2026-09-28T12:34:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1048 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:45 dist-test-slave-g31s krb5kdc[26279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:45 dist-test-slave-g31s krb5kdc[26279](info): set up 2 sockets
Sep 28 12:34:45 dist-test-slave-g31s krb5kdc[26279](info): commencing operation
krb5kdc: starting...
W20260928 12:34:46.029179 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:34:46 dist-test-slave-g31s krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:34:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:34:46.140837 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33253
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:33253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:46.224619 26292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:46.224838 26292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:46.224867 26292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:46.226049 26292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:46.226094 26292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:46.226111 26292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:46.226126 26292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:46.226142 26292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:46.227640 26292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:33253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:46.227893 26292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:46.228098 26292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:46.232867 26298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:46.232896 26292 server_base.cc:1034] running on GCE node
W20260928 12:34:46.232867 26297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:46.232920 26300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:46.233330 26292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:46.233647 26292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:46.234784 26292 hybrid_clock.cc:648] HybridClock initialized: now 1790598886234760 us; error 41 us; skew 500 ppm
Sep 28 12:34:46 dist-test-slave-g31s krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598886, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:46.237365 26292 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:34:46.237917 26292 webserver.cc:460] Webserver started at http://127.8.13.62:43121/ using document root <none> and password file <none>
I20260928 12:34:46.238137 26292 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:46.238188 26292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:46.238317 26292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:46.239522 26292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "3d1189a3f5c54b9381a65d1acc9b7571"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "48beb00a2acc2368b423a049db2edae6"
server_key_iv: "466966f624ea650ef2aa2ba7a15e3a3d"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.239871 26292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "3d1189a3f5c54b9381a65d1acc9b7571"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "48beb00a2acc2368b423a049db2edae6"
server_key_iv: "466966f624ea650ef2aa2ba7a15e3a3d"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.241169 26292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:34:46.241851 26307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:46.242033 26292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:46.242122 26292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "3d1189a3f5c54b9381a65d1acc9b7571"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "48beb00a2acc2368b423a049db2edae6"
server_key_iv: "466966f624ea650ef2aa2ba7a15e3a3d"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.242192 26292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:46.255255 26292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:46.257544 26292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:46.257699 26292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:46.261641 26292 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:33253
I20260928 12:34:46.261670 26359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:33253 every 8 connection(s)
I20260928 12:34:46.262092 26292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:46.263027 26360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:46.265718 26360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571: Bootstrap starting.
I20260928 12:34:46.266372 26360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:46.266681 26360 log.cc:824] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:46.267428 26360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571: No bootstrap required, opened a new log
I20260928 12:34:46.269104 26360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1189a3f5c54b9381a65d1acc9b7571" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33253 } }
I20260928 12:34:46.269258 26360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:46.269279 26360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1189a3f5c54b9381a65d1acc9b7571, State: Initialized, Role: FOLLOWER
I20260928 12:34:46.269448 26360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1189a3f5c54b9381a65d1acc9b7571" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33253 } }
I20260928 12:34:46.269526 26360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:46.269550 26360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:46.269598 26360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:46.270120 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 26292
I20260928 12:34:46.270239 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:46.270223 26360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1189a3f5c54b9381a65d1acc9b7571" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33253 } }
I20260928 12:34:46.270327 26360 leader_election.cc:302] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d1189a3f5c54b9381a65d1acc9b7571; no voters:
I20260928 12:34:46.270453 8244 external_mini_cluster.cc:1428] Setting key 62949a2000e609429e098a63f104f0cc
I20260928 12:34:46.270514 26360 leader_election.cc:288] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:46.270551 26363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:46.270709 26363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1189a3f5c54b9381a65d1acc9b7571, State: Running, Role: LEADER
I20260928 12:34:46.270802 26360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:46.270815 26363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1189a3f5c54b9381a65d1acc9b7571" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33253 } }
I20260928 12:34:46.271180 26363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d1189a3f5c54b9381a65d1acc9b7571. Latest consensus state: current_term: 1 leader_uuid: "3d1189a3f5c54b9381a65d1acc9b7571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1189a3f5c54b9381a65d1acc9b7571" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33253 } } }
I20260928 12:34:46.271255 26363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:46.271438 26371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:46.271553 26364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d1189a3f5c54b9381a65d1acc9b7571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1189a3f5c54b9381a65d1acc9b7571" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33253 } } }
I20260928 12:34:46.271631 26364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:46.272527 26371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:46.274065 26371 catalog_manager.cc:1347] Generated new cluster ID: 9d5e51358a784368ab7b2b9fa8f7cc88
I20260928 12:34:46.274120 26371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:34:46 dist-test-slave-g31s krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:46.282934 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.271119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.8.13.62:33253)
0928 12:34:46.271400 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:46.271410 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:46.271447 (+ 37us) server_negotiation.cc:407] Connection header received
0928 12:34:46.271576 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:46.271583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:46.271651 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:46.272447 (+ 796us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:46.272777 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.274141 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.275202 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.275909 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.276029 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.278931 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:46.278937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:46.278937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:46.278954 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:46.281087 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:46.281555 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:46.281559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:46.281559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:46.281690 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:46.281910 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:46.281911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:46.281911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:46.282254 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:46.282416 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:46.282512 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 12:34:46.282664 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":82,"threads_started":1}
I20260928 12:34:46.310926 26371 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:46.311491 26371 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:46.315582 26371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571: Generated new TSK 0
I20260928 12:34:46.315757 26371 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:34:46.344256 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33253
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:46.426151 26386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:46.426379 26386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:46.426412 26386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:46.427559 26386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:46.427601 26386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:46.427636 26386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:34:46.429083 26386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:46.429322 26386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:46.429524 26386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:46.434556 26394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:46.434633 26391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:46.434547 26392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:46.434854 26386 server_base.cc:1034] running on GCE node
I20260928 12:34:46.435053 26386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:46.435278 26386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:46.436414 26386 hybrid_clock.cc:648] HybridClock initialized: now 1790598886436400 us; error 31 us; skew 500 ppm
Sep 28 12:34:46 dist-test-slave-g31s krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598886, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:46.439221 26386 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:34:46.439690 26386 webserver.cc:460] Webserver started at http://127.8.13.1:37973/ using document root <none> and password file <none>
I20260928 12:34:46.439909 26386 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:46.439953 26386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:46.440096 26386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:46.441195 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "335f5f3549734becadca38a66bbc61d6"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "5b9a1b2c5586fd63669ee8acd594ec90"
server_key_iv: "b2e1048902a5235371b3c00ae1061146"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.441534 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "335f5f3549734becadca38a66bbc61d6"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "5b9a1b2c5586fd63669ee8acd594ec90"
server_key_iv: "b2e1048902a5235371b3c00ae1061146"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.442922 26386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:34:46.443748 26401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:46.443954 26386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:46.444046 26386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "335f5f3549734becadca38a66bbc61d6"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "5b9a1b2c5586fd63669ee8acd594ec90"
server_key_iv: "b2e1048902a5235371b3c00ae1061146"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.444109 26386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:46.459467 26386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:46.461560 26386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:46.461686 26386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:46.461931 26386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:46.462234 26386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:46.462270 26386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:46.462296 26386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:46.462313 26386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:46.468051 26386 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:41799
I20260928 12:34:46.468111 26514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:41799 every 8 connection(s)
I20260928 12:34:46.468426 26386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:34:46.469632 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 26386
I20260928 12:34:46.469715 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:34:46.469933 8244 external_mini_cluster.cc:1428] Setting key 71b031067facd7494cb4c286ffbec6ba
Sep 28 12:34:46 dist-test-slave-g31s krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598886, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
I20260928 12:34:46.478389 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.469162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:52685 (local address 127.8.13.62:33253)
0928 12:34:46.469216 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:46.469217 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:46.469435 (+ 218us) server_negotiation.cc:407] Connection header received
0928 12:34:46.470084 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:46.470086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:46.470102 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:46.470167 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:46.471314 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.472060 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.473098 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.473308 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.473364 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.475674 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:46.475675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:46.475676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:46.475683 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:46.476985 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:46.477360 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:46.477362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:46.477362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:46.477463 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:46.477692 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:46.477693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:46.477694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:46.477854 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:46.477921 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:46.478191 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 12:34:46.478292 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:34:46.478554 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.469134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33253 (local address 127.8.13.1:52685)
0928 12:34:46.469311 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:46.469354 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:46.469946 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:46.470320 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:46.470323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:46.471007 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:46.471211 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.471216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.472154 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.472155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.473018 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.473021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.473379 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.473380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.473507 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.474052 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:46.474061 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:46.475583 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:46.477063 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:46.477070 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:46.477073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:46.477290 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:46.477523 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:46.477524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:46.477524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:46.477634 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:46.477921 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:46.477928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:46.478071 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 12:34:46.478162 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 12:34:46.478329 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260928 12:34:46.479048 26515 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33253
I20260928 12:34:46.479148 26515 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:46.479346 26515 heartbeater.cc:511] Master 127.8.13.62:33253 requested a full tablet report, sending...
I20260928 12:34:46.479873 26324 ts_manager.cc:194] Registered new tserver with Master: 335f5f3549734becadca38a66bbc61d6 (127.8.13.1:41799)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
I20260928 12:34:46.480517 26324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:52685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:34:46.497262 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33253
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:46.578923 26520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:46.579161 26520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:46.579188 26520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:46.580447 26520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:46.580520 26520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:46.580550 26520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:34:46.582278 26520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:46.582556 26520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:46.582784 26520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:46.587759 26528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:46.587759 26525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:46.587787 26526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:46.587852 26520 server_base.cc:1034] running on GCE node
I20260928 12:34:46.588196 26520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:46.588438 26520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:46.589600 26520 hybrid_clock.cc:648] HybridClock initialized: now 1790598886589583 us; error 30 us; skew 500 ppm
Sep 28 12:34:46 dist-test-slave-g31s krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598886, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:46.592259 26520 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:34:46.592744 26520 webserver.cc:460] Webserver started at http://127.8.13.2:33627/ using document root <none> and password file <none>
I20260928 12:34:46.592959 26520 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:46.593016 26520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:46.593145 26520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:46.594293 26520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "227664cc840a4a458ddfc149739fa97d"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "db7f41006eb681f2dad68eb68c1ed39f"
server_key_iv: "6de6fc32bdc6f244034272676a314475"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.594633 26520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "227664cc840a4a458ddfc149739fa97d"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "db7f41006eb681f2dad68eb68c1ed39f"
server_key_iv: "6de6fc32bdc6f244034272676a314475"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.596067 26520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:46.596876 26535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:46.597035 26520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:46.597129 26520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "227664cc840a4a458ddfc149739fa97d"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "db7f41006eb681f2dad68eb68c1ed39f"
server_key_iv: "6de6fc32bdc6f244034272676a314475"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.597203 26520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:46.609145 26520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:46.611299 26520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:46.611434 26520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:46.611658 26520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:46.611954 26520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:46.612000 26520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:46.612032 26520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:46.612056 26520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:46.617708 26520 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:42745
I20260928 12:34:46.617767 26648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:42745 every 8 connection(s)
I20260928 12:34:46.618125 26520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:34:46.622272 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 26520
I20260928 12:34:46.622397 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:34:46.622575 8244 external_mini_cluster.cc:1428] Setting key f1556b2a449cabd8f0fca49ca634f9b5
Sep 28 12:34:46 dist-test-slave-g31s krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598886, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:46.627082 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.618747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:43507 (local address 127.8.13.62:33253)
0928 12:34:46.618797 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:46.618800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:46.619008 (+ 208us) server_negotiation.cc:407] Connection header received
0928 12:34:46.619522 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:46.619523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:46.619537 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:46.619587 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:46.620486 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.621005 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.621989 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.622220 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.622291 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.624567 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:46.624569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:46.624569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:46.624577 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:46.625877 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:46.626229 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:46.626231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:46.626232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:46.626287 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:46.626489 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:46.626490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:46.626491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:46.626603 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:46.626657 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:46.626901 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 12:34:46.626995 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:34:46.627224 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.618723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33253 (local address 127.8.13.2:43507)
0928 12:34:46.618891 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:46.618926 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:46.619424 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:46.619653 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:46.619656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:46.620204 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:46.620392 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.620397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.621087 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.621088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.621908 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.621910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.622314 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.622316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.622477 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.622936 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:46.622945 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:46.624450 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:46.625943 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:46.625951 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:46.625953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:46.626162 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:46.626339 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:46.626340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:46.626340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:46.626435 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:46.626654 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:46.626661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:46.626790 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 12:34:46.626874 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 12:34:46.627024 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260928 12:34:46.627629 26649 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33253
I20260928 12:34:46.627731 26649 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:46.627928 26649 heartbeater.cc:511] Master 127.8.13.62:33253 requested a full tablet report, sending...
I20260928 12:34:46.628302 26324 ts_manager.cc:194] Registered new tserver with Master: 227664cc840a4a458ddfc149739fa97d (127.8.13.2:42745)
I20260928 12:34:46.628662 26324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:43507
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:34:46.648263 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33253
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:46.727562 26654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:46.727753 26654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:46.727775 26654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:46.728895 26654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:46.728935 26654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:46.728961 26654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:34:46.730441 26654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:46.730649 26654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:46.730849 26654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:46.735775 26660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:46.735760 26659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:46.735782 26662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:46.736011 26654 server_base.cc:1034] running on GCE node
I20260928 12:34:46.736179 26654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:46.736390 26654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:46.737532 26654 hybrid_clock.cc:648] HybridClock initialized: now 1790598886737520 us; error 25 us; skew 500 ppm
Sep 28 12:34:46 dist-test-slave-g31s krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598886, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:46.740093 26654 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:34:46.740559 26654 webserver.cc:460] Webserver started at http://127.8.13.3:43683/ using document root <none> and password file <none>
I20260928 12:34:46.740763 26654 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:46.740804 26654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:46.740907 26654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:46.742043 26654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "6710b717817a4cf684c9018283b75c15"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "2b44f7740709adc5a0245d018403b995"
server_key_iv: "2451b605fe0bbbaa08dd7067705d3b16"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.742388 26654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "6710b717817a4cf684c9018283b75c15"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "2b44f7740709adc5a0245d018403b995"
server_key_iv: "2451b605fe0bbbaa08dd7067705d3b16"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.743593 26654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:34:46.744293 26669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:46.744488 26654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:46.744596 26654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "6710b717817a4cf684c9018283b75c15"
format_stamp: "Formatted at 2026-09-28 12:34:46 on dist-test-slave-g31s"
server_key: "2b44f7740709adc5a0245d018403b995"
server_key_iv: "2451b605fe0bbbaa08dd7067705d3b16"
server_key_version: "encryptionkey@0"
I20260928 12:34:46.744673 26654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:46.767740 26654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:46.769601 26654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:46.769731 26654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:46.769992 26654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:46.770320 26654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:46.770364 26654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:46.770396 26654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:46.770419 26654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:46.776484 26654 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:37239
I20260928 12:34:46.776527 26782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:37239 every 8 connection(s)
I20260928 12:34:46.776870 26654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:34:46 dist-test-slave-g31s krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598886, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:46.783370 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 26654
I20260928 12:34:46.783490 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:34:46.783704 8244 external_mini_cluster.cc:1428] Setting key 016edd5e2d2387ef8a0e772bae2993bf
I20260928 12:34:46.786482 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.777651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:59179 (local address 127.8.13.62:33253)
0928 12:34:46.777710 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:46.777712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:46.777930 (+ 218us) server_negotiation.cc:407] Connection header received
0928 12:34:46.778425 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:46.778426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:46.778437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:46.778504 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:46.779533 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.780064 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.781101 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.781302 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.781371 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.783724 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:46.783726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:46.783726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:46.783733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:46.785250 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:46.785646 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:46.785647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:46.785648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:46.785703 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:46.785933 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:46.785934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:46.785934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:46.786050 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:46.786102 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:46.786354 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 12:34:46.786416 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:34:46.786653 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.777609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33253 (local address 127.8.13.3:59179)
0928 12:34:46.777785 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:46.777852 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:46.778345 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:46.778568 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:46.778572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:46.779200 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:46.779403 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.779408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.780169 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.780170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.781014 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.781016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.781367 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.781368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.781471 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.782066 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:46.782075 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:46.783619 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:46.785331 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:46.785337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:46.785339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:46.785587 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:46.785746 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:46.785747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:46.785747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:46.785885 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:46.786101 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:46.786106 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:46.786235 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 12:34:46.786319 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 12:34:46.786454 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260928 12:34:46.787084 26783 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33253
I20260928 12:34:46.787173 26783 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:46.787334 26783 heartbeater.cc:511] Master 127.8.13.62:33253 requested a full tablet report, sending...
I20260928 12:34:46.787655 26324 ts_manager.cc:194] Registered new tserver with Master: 6710b717817a4cf684c9018283b75c15 (127.8.13.3:37239)
I20260928 12:34:46.788049 26324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:59179
I20260928 12:34:46.795634 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:34:46.803289 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.796918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.8.13.62:33253)
0928 12:34:46.796988 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:46.796990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:46.797081 (+ 91us) server_negotiation.cc:407] Connection header received
0928 12:34:46.797138 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:46.797139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:46.797149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:46.797190 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:46.797755 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.798293 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.799041 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.799233 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.799287 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.800929 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:46.800930 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:46.800930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:46.800938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:46.802116 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:46.802504 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:46.802505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:46.802506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:46.802564 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:46.802750 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:46.802751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:46.802751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:46.802892 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:46.802993 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:46.803133 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 12:34:46.803195 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:34:46.804832 26324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:34:46.806447 26324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:34:46.812420 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.808669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.8.13.2:42745)
0928 12:34:46.808859 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:46.808864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:46.808873 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:34:46.808908 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:46.808910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:46.808924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:46.808991 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:46.809184 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.810092 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.811493 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.812046 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.812106 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.812165 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:46.812306 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 12:34:46.812341 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260928 12:34:46.812409 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.808354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:42745 (local address 127.0.0.1:37612)
0928 12:34:46.808653 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:46.808693 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:46.808755 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:46.809001 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:46.809004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:46.809007 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:46.809100 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.809103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.810177 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.810178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.811410 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.811412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.812118 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.812120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.812210 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.812223 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:34:46.812284 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 12:34:46.812338 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260928 12:34:46.812775 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.808473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.8.13.1:41799)
0928 12:34:46.808655 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:46.808660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:46.808883 (+ 223us) server_negotiation.cc:407] Connection header received
0928 12:34:46.808951 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:46.808953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:46.808968 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:46.809032 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:46.809267 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.810371 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.811611 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.812328 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.812397 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.812447 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:46.812655 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 12:34:46.812701 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260928 12:34:46.812898 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.808658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.8.13.3:37239)
0928 12:34:46.808815 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:46.808819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:46.809201 (+ 382us) server_negotiation.cc:407] Connection header received
0928 12:34:46.809248 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:46.809251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:46.809265 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:46.809340 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:46.809545 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.810585 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.811668 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.812333 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.812397 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.812447 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:46.812766 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 12:34:46.812823 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260928 12:34:46.812731 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.808553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:37239 (local address 127.0.0.1:46630)
0928 12:34:46.809084 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:46.809094 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:46.809141 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:46.809423 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:46.809424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:46.809426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:46.809461 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.809462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.810667 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.810668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.811581 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.811584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.812484 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.812484 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.812550 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.812555 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:34:46.812602 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:34:46.812640 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":5,"thread_start_us":41,"threads_started":1}
I20260928 12:34:46.812721 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.808355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:41799 (local address 127.0.0.1:55378)
0928 12:34:46.808822 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:46.808832 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:46.808889 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:46.809033 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:46.809034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:46.809036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:46.809133 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.809135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.810473 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.810474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.811540 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.811542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.812482 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.812483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.812550 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.812556 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:34:46.812631 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:34:46.812660 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":21,"thread_start_us":72,"threads_started":1}
I20260928 12:34:46.813414 26583 tablet_service.cc:1467] Processing CreateTablet for tablet aecf158576244da19a2f947ebd2a7d30 (DEFAULT_TABLE table=test-table [id=fbd5b4d185e64a508ae8689726d84ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:46.813657 26717 tablet_service.cc:1467] Processing CreateTablet for tablet aecf158576244da19a2f947ebd2a7d30 (DEFAULT_TABLE table=test-table [id=fbd5b4d185e64a508ae8689726d84ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:46.813777 26583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecf158576244da19a2f947ebd2a7d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:46.813963 26717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecf158576244da19a2f947ebd2a7d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:46.817009 26798 tablet_bootstrap.cc:492] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15: Bootstrap starting.
I20260928 12:34:46.817365 26449 tablet_service.cc:1467] Processing CreateTablet for tablet aecf158576244da19a2f947ebd2a7d30 (DEFAULT_TABLE table=test-table [id=fbd5b4d185e64a508ae8689726d84ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:46.817582 26798 tablet_bootstrap.cc:654] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:46.817596 26449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecf158576244da19a2f947ebd2a7d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:46.818024 26798 log.cc:824] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:46.818343 26797 tablet_bootstrap.cc:492] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d: Bootstrap starting.
I20260928 12:34:46.818686 26798 tablet_bootstrap.cc:492] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15: No bootstrap required, opened a new log
I20260928 12:34:46.818738 26798 ts_tablet_manager.cc:1397] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:34:46.820032 26800 tablet_bootstrap.cc:492] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6: Bootstrap starting.
I20260928 12:34:46.820278 26798 raft_consensus.cc:348] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335f5f3549734becadca38a66bbc61d6" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41799 } } peers { permanent_uuid: "227664cc840a4a458ddfc149739fa97d" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42745 } } peers { permanent_uuid: "6710b717817a4cf684c9018283b75c15" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37239 } }
I20260928 12:34:46.820426 26798 raft_consensus.cc:374] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:46.820447 26798 raft_consensus.cc:729] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6710b717817a4cf684c9018283b75c15, State: Initialized, Role: FOLLOWER
I20260928 12:34:46.820530 26798 consensus_queue.cc:260] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335f5f3549734becadca38a66bbc61d6" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41799 } } peers { permanent_uuid: "227664cc840a4a458ddfc149739fa97d" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42745 } } peers { permanent_uuid: "6710b717817a4cf684c9018283b75c15" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37239 } }
I20260928 12:34:46.820619 26800 tablet_bootstrap.cc:654] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:46.820770 26798 ts_tablet_manager.cc:1428] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:34:46.820827 26783 heartbeater.cc:503] Master 127.8.13.62:33253 was elected leader, sending a full tablet report...
I20260928 12:34:46.821147 26800 log.cc:824] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:46.821369 26797 tablet_bootstrap.cc:654] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:46.821923 26800 tablet_bootstrap.cc:492] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6: No bootstrap required, opened a new log
I20260928 12:34:46.821905 26797 log.cc:824] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:46.821990 26800 ts_tablet_manager.cc:1397] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:34:46.822520 26797 tablet_bootstrap.cc:492] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d: No bootstrap required, opened a new log
I20260928 12:34:46.822571 26797 ts_tablet_manager.cc:1397] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 12:34:46.823377 26800 raft_consensus.cc:348] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335f5f3549734becadca38a66bbc61d6" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41799 } } peers { permanent_uuid: "227664cc840a4a458ddfc149739fa97d" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42745 } } peers { permanent_uuid: "6710b717817a4cf684c9018283b75c15" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37239 } }
I20260928 12:34:46.823549 26800 raft_consensus.cc:374] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:46.823596 26800 raft_consensus.cc:729] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 335f5f3549734becadca38a66bbc61d6, State: Initialized, Role: FOLLOWER
I20260928 12:34:46.823720 26800 consensus_queue.cc:260] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335f5f3549734becadca38a66bbc61d6" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41799 } } peers { permanent_uuid: "227664cc840a4a458ddfc149739fa97d" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42745 } } peers { permanent_uuid: "6710b717817a4cf684c9018283b75c15" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37239 } }
I20260928 12:34:46.823990 26800 ts_tablet_manager.cc:1428] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:34:46.823961 26797 raft_consensus.cc:348] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335f5f3549734becadca38a66bbc61d6" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41799 } } peers { permanent_uuid: "227664cc840a4a458ddfc149739fa97d" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42745 } } peers { permanent_uuid: "6710b717817a4cf684c9018283b75c15" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37239 } }
I20260928 12:34:46.824118 26515 heartbeater.cc:503] Master 127.8.13.62:33253 was elected leader, sending a full tablet report...
I20260928 12:34:46.824131 26797 raft_consensus.cc:374] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:46.824170 26797 raft_consensus.cc:729] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227664cc840a4a458ddfc149739fa97d, State: Initialized, Role: FOLLOWER
I20260928 12:34:46.824267 26797 consensus_queue.cc:260] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335f5f3549734becadca38a66bbc61d6" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41799 } } peers { permanent_uuid: "227664cc840a4a458ddfc149739fa97d" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42745 } } peers { permanent_uuid: "6710b717817a4cf684c9018283b75c15" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37239 } }
I20260928 12:34:46.824520 26797 ts_tablet_manager.cc:1428] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:34:46.824545 26649 heartbeater.cc:503] Master 127.8.13.62:33253 was elected leader, sending a full tablet report...
W20260928 12:34:46.868664 26650 tablet.cc:2369] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:34:46.912673 26801 raft_consensus.cc:482] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:34:46.912828 26801 raft_consensus.cc:504] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335f5f3549734becadca38a66bbc61d6" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41799 } } peers { permanent_uuid: "227664cc840a4a458ddfc149739fa97d" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42745 } } peers { permanent_uuid: "6710b717817a4cf684c9018283b75c15" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37239 } }
I20260928 12:34:46.913275 26801 leader_election.cc:288] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 335f5f3549734becadca38a66bbc61d6 (127.8.13.1:41799), 227664cc840a4a458ddfc149739fa97d (127.8.13.2:42745)
I20260928 12:34:46.916970 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.913707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:40065 (local address 127.8.13.2:42745)
0928 12:34:46.913754 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:46.913756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:46.913936 (+ 180us) server_negotiation.cc:407] Connection header received
0928 12:34:46.914018 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:46.914019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:46.914031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:46.914077 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:46.914205 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.914980 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.916273 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.916660 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.916724 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.916734 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:46.916867 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 12:34:46.916904 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:34:46.916970 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.913547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:57533 (local address 127.8.13.1:41799)
0928 12:34:46.913593 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:46.913596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:46.913786 (+ 190us) server_negotiation.cc:407] Connection header received
0928 12:34:46.913901 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:46.913902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:46.913917 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:46.913968 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:46.914233 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.914958 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.916272 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.916653 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.916712 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.916724 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:46.916853 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 12:34:46.916894 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:34:46.916978 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.913476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:42745 (local address 127.8.13.3:40065)
0928 12:34:46.913880 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:46.913890 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:46.913946 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:46.914075 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:46.914076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:46.914077 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:46.914138 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.914140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.915065 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.915066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.916188 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.916191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.916734 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.916734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.916807 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.916809 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:34:46.916856 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:34:46.916906 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":35,"thread_start_us":105,"threads_started":1}
I20260928 12:34:46.916985 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.913439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:41799 (local address 127.8.13.3:57533)
0928 12:34:46.913716 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:46.913726 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:46.913823 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:46.914018 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:46.914019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:46.914021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:46.914121 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.914124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.915034 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.915035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.916189 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:46.916192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.916725 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.916726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:46.916793 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.916797 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:34:46.916850 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:34:46.916890 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":39}
I20260928 12:34:46.917285 26603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecf158576244da19a2f947ebd2a7d30" candidate_uuid: "6710b717817a4cf684c9018283b75c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227664cc840a4a458ddfc149739fa97d" is_pre_election: true
I20260928 12:34:46.917430 26603 raft_consensus.cc:2454] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6710b717817a4cf684c9018283b75c15 in term 0.
I20260928 12:34:46.917549 26469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecf158576244da19a2f947ebd2a7d30" candidate_uuid: "6710b717817a4cf684c9018283b75c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "335f5f3549734becadca38a66bbc61d6" is_pre_election: true
I20260928 12:34:46.917634 26673 leader_election.cc:302] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227664cc840a4a458ddfc149739fa97d, 6710b717817a4cf684c9018283b75c15; no voters:
I20260928 12:34:46.917681 26469 raft_consensus.cc:2454] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6710b717817a4cf684c9018283b75c15 in term 0.
I20260928 12:34:46.917758 26801 raft_consensus.cc:2781] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:34:46.917831 26801 raft_consensus.cc:482] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:34:46.917857 26801 raft_consensus.cc:3037] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:46.918512 26801 raft_consensus.cc:504] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335f5f3549734becadca38a66bbc61d6" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41799 } } peers { permanent_uuid: "227664cc840a4a458ddfc149739fa97d" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42745 } } peers { permanent_uuid: "6710b717817a4cf684c9018283b75c15" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37239 } }
I20260928 12:34:46.918627 26801 leader_election.cc:288] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [CANDIDATE]: Term 1 election: Requested vote from peers 335f5f3549734becadca38a66bbc61d6 (127.8.13.1:41799), 227664cc840a4a458ddfc149739fa97d (127.8.13.2:42745)
I20260928 12:34:46.918798 26469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecf158576244da19a2f947ebd2a7d30" candidate_uuid: "6710b717817a4cf684c9018283b75c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "335f5f3549734becadca38a66bbc61d6"
I20260928 12:34:46.918881 26469 raft_consensus.cc:3037] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:46.918867 26603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecf158576244da19a2f947ebd2a7d30" candidate_uuid: "6710b717817a4cf684c9018283b75c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227664cc840a4a458ddfc149739fa97d"
I20260928 12:34:46.918954 26603 raft_consensus.cc:3037] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:46.919592 26469 raft_consensus.cc:2454] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6710b717817a4cf684c9018283b75c15 in term 1.
I20260928 12:34:46.919659 26603 raft_consensus.cc:2454] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6710b717817a4cf684c9018283b75c15 in term 1.
I20260928 12:34:46.919754 26670 leader_election.cc:302] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 335f5f3549734becadca38a66bbc61d6, 6710b717817a4cf684c9018283b75c15; no voters:
I20260928 12:34:46.919847 26801 raft_consensus.cc:2781] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:46.919988 26801 raft_consensus.cc:686] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 1 LEADER]: Becoming Leader. State: Replica: 6710b717817a4cf684c9018283b75c15, State: Running, Role: LEADER
I20260928 12:34:46.920056 26801 consensus_queue.cc:237] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335f5f3549734becadca38a66bbc61d6" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41799 } } peers { permanent_uuid: "227664cc840a4a458ddfc149739fa97d" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42745 } } peers { permanent_uuid: "6710b717817a4cf684c9018283b75c15" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37239 } }
I20260928 12:34:46.920780 26323 catalog_manager.cc:5495] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6710b717817a4cf684c9018283b75c15 (127.8.13.3). New cstate: current_term: 1 leader_uuid: "6710b717817a4cf684c9018283b75c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335f5f3549734becadca38a66bbc61d6" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "227664cc840a4a458ddfc149739fa97d" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6710b717817a4cf684c9018283b75c15" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260928 12:34:46.927939 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.925874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.8.13.3:37239)
0928 12:34:46.925943 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:46.925944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:46.925952 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:34:46.926019 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:46.926020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:46.926036 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:46.926090 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:34:46.926181 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.926613 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.927341 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:46.927522 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:46.927569 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:46.927675 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:34:46.927751 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:34:46.927813 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:46.927844 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 12:34:46.927851 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 12:34:46.929533 26469 raft_consensus.cc:1261] T aecf158576244da19a2f947ebd2a7d30 P 335f5f3549734becadca38a66bbc61d6 [term 1 FOLLOWER]: Refusing update from remote peer 6710b717817a4cf684c9018283b75c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:46.929533 26603 raft_consensus.cc:1261] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d [term 1 FOLLOWER]: Refusing update from remote peer 6710b717817a4cf684c9018283b75c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:46.929847 26807 consensus_queue.cc:1035] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227664cc840a4a458ddfc149739fa97d" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:46.929977 26801 consensus_queue.cc:1035] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "335f5f3549734becadca38a66bbc61d6" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:46.932511 26812 mvcc.cc:205] Tried to move back new op lower bound from 7334293040861048832 to 7334293040825004032. Current Snapshot: MvccSnapshot[applied={T|T < 7334293040861048832}]
I20260928 12:34:46.933670 26811 mvcc.cc:205] Tried to move back new op lower bound from 7334293040861048832 to 7334293040825004032. Current Snapshot: MvccSnapshot[applied={T|T < 7334293040861048832}]
I20260928 12:34:46.933980 26323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:46.934258 26323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:46.934983 26323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d1189a3f5c54b9381a65d1acc9b7571: Sending DeleteTablet for 3 replicas of tablet aecf158576244da19a2f947ebd2a7d30
I20260928 12:34:46.935326 26583 tablet_service.cc:1514] Processing DeleteTablet for tablet aecf158576244da19a2f947ebd2a7d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:46 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:37612
I20260928 12:34:46.935374 26717 tablet_service.cc:1514] Processing DeleteTablet for tablet aecf158576244da19a2f947ebd2a7d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:46 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:46630
I20260928 12:34:46.935348 26449 tablet_service.cc:1514] Processing DeleteTablet for tablet aecf158576244da19a2f947ebd2a7d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:46 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:55378
I20260928 12:34:46.935542 26820 tablet_replica.cc:330] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d: stopping tablet replica
I20260928 12:34:46.935595 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 26386
I20260928 12:34:46.935636 26820 raft_consensus.cc:2229] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:34:46.937929 26822 tablet_replica.cc:330] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15: stopping tablet replica
I20260928 12:34:46.938042 26822 raft_consensus.cc:2229] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:34:46.938088 26820 raft_consensus.cc:2258] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:46.938153 26822 raft_consensus.cc:2258] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:46.938638 26822 ts_tablet_manager.cc:1905] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:46.940510 26822 ts_tablet_manager.cc:1918] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:46.940645 26822 log.cc:1196] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/aecf158576244da19a2f947ebd2a7d30
I20260928 12:34:46.940719 26820 ts_tablet_manager.cc:1905] T aecf158576244da19a2f947ebd2a7d30 P 227664cc840a4a458ddfc149739fa97d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:46.941069 26822 ts_tablet_manager.cc:1939] T aecf158576244da19a2f947ebd2a7d30 P 6710b717817a4cf684c9018283b75c15: Deleting consensus metadata
I20260928 12:34:46.941567 26308 catalog_manager.cc:4841] TS 6710b717817a4cf684c9018283b75c15 (127.8.13.3:37239): tablet aecf158576244da19a2f947ebd2a7d30 (table test-table [id=fbd5b4d185e64a508ae8689726d84ac3]) successfully deleted
W20260928 12:34:46.941736 26308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:41799 (error 108)
I20260928 12:34:46.941977 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 26520
I20260928 12:34:46.945883 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.945652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:41799 (local address 127.0.0.1:55392)
0928 12:34:46.945717 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:46.945829 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:41799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 12:34:46.946118 26308 catalog_manager.cc:4584] TS 335f5f3549734becadca38a66bbc61d6 (127.8.13.1:41799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aecf158576244da19a2f947ebd2a7d30: Network error: Client connection negotiation failed: client connection to 127.8.13.1:41799: connect: Connection refused (error 111)
I20260928 12:34:46.947207 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:46.947081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:42745 (local address 127.0.0.1:37618)
0928 12:34:46.947141 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:46.947169 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.2:42745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 12:34:46.947225 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 26654
I20260928 12:34:46.951925 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 26292
2026-09-28T12:34:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1869 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T12:34:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:46Z Disabled control of system clock
I20260928 12:34:46.978991 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:40601
--webserver_interface=127.8.13.62
--webserver_port=0
--builtin_ntp_servers=127.8.13.20:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:40601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:34:47.058478 26827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:47.058723 26827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:47.058745 26827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:47.060031 26827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:47.060086 26827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:47.060097 26827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:47.060108 26827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:47.060120 26827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:47.061573 26827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:40601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:40601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:47.061865 26827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:47.062163 26827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:47.066933 26832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:47.066933 26835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:47.066951 26833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:47.066989 26827 server_base.cc:1034] running on GCE node
I20260928 12:34:47.067552 26827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:47.067888 26827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:47.069041 26827 hybrid_clock.cc:648] HybridClock initialized: now 1790598887069023 us; error 40 us; skew 500 ppm
I20260928 12:34:47.070402 26827 webserver.cc:460] Webserver started at http://127.8.13.62:43251/ using document root <none> and password file <none>
I20260928 12:34:47.070644 26827 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:47.070695 26827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:47.070827 26827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:47.071905 26827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "a93a6a13d5a84c50b4200955c8591d27"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "51f5c2f9a6032f783199631fd6c00ebb"
server_key_iv: "4cf464de83991d5c59159a07ed329740"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.072288 26827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "a93a6a13d5a84c50b4200955c8591d27"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "51f5c2f9a6032f783199631fd6c00ebb"
server_key_iv: "4cf464de83991d5c59159a07ed329740"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.073613 26827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:34:47.074411 26841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:47.074579 26827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:47.074662 26827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "a93a6a13d5a84c50b4200955c8591d27"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "51f5c2f9a6032f783199631fd6c00ebb"
server_key_iv: "4cf464de83991d5c59159a07ed329740"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.074733 26827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:47.084265 26827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:47.084621 26827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:47.084764 26827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:47.088579 26827 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:40601
I20260928 12:34:47.088615 26893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:40601 every 8 connection(s)
I20260928 12:34:47.088963 26827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:47.089919 26894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:47.092247 26894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27: Bootstrap starting.
I20260928 12:34:47.092760 26894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:47.092998 26894 log.cc:824] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:47.093360 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 26827
I20260928 12:34:47.093459 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:47.093657 8244 external_mini_cluster.cc:1428] Setting key 7bdfe8d38c2905521bb34935fcea2491
I20260928 12:34:47.093719 26894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27: No bootstrap required, opened a new log
I20260928 12:34:47.095607 26894 raft_consensus.cc:348] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93a6a13d5a84c50b4200955c8591d27" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40601 } }
I20260928 12:34:47.095747 26894 raft_consensus.cc:374] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:47.095769 26894 raft_consensus.cc:729] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93a6a13d5a84c50b4200955c8591d27, State: Initialized, Role: FOLLOWER
I20260928 12:34:47.095871 26894 consensus_queue.cc:260] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93a6a13d5a84c50b4200955c8591d27" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40601 } }
I20260928 12:34:47.095954 26894 raft_consensus.cc:388] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:47.095981 26894 raft_consensus.cc:482] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:47.096027 26894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:47.096596 26894 raft_consensus.cc:504] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93a6a13d5a84c50b4200955c8591d27" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40601 } }
I20260928 12:34:47.096684 26894 leader_election.cc:302] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a93a6a13d5a84c50b4200955c8591d27; no voters:
I20260928 12:34:47.096844 26894 leader_election.cc:288] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:47.096890 26899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:47.097018 26899 raft_consensus.cc:686] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [term 1 LEADER]: Becoming Leader. State: Replica: a93a6a13d5a84c50b4200955c8591d27, State: Running, Role: LEADER
I20260928 12:34:47.097103 26894 sys_catalog.cc:564] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:47.097169 26899 consensus_queue.cc:237] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93a6a13d5a84c50b4200955c8591d27" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40601 } }
I20260928 12:34:47.097554 26899 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a93a6a13d5a84c50b4200955c8591d27. Latest consensus state: current_term: 1 leader_uuid: "a93a6a13d5a84c50b4200955c8591d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93a6a13d5a84c50b4200955c8591d27" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40601 } } }
I20260928 12:34:47.097632 26899 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:47.097743 26901 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a93a6a13d5a84c50b4200955c8591d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93a6a13d5a84c50b4200955c8591d27" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 40601 } } }
I20260928 12:34:47.097855 26905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:47.097913 26901 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:47.098454 26905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:47.100029 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.094222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.8.13.62:40601)
0928 12:34:47.094440 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:47.094449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:47.094473 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:34:47.094939 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:47.094946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:47.095008 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:47.095139 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:47.095353 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.096732 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.097742 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.098371 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.098468 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.099364 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:47.099368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:47.099368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:47.099534 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:47.099571 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:47.099582 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:47.099636 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:47.099712 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 12:34:47.099797 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260928 12:34:47.100392 26905 catalog_manager.cc:1347] Generated new cluster ID: e9b209f46c8b4824881a5ada43f37018
I20260928 12:34:47.100441 26905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:47.135121 26905 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:47.135663 26905 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:47.143496 26905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27: Generated new TSK 0
I20260928 12:34:47.143697 26905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:34:47.154211 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40601
--builtin_ntp_servers=127.8.13.20:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:34:47.234495 26918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:47.234829 26918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:47.234879 26918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:47.236135 26918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:47.236186 26918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:47.236213 26918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:34:47.237663 26918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:47.237957 26918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:47.238284 26918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:34:47.243520 26918 server_base.cc:1034] running on GCE node
W20260928 12:34:47.243537 26924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:47.243546 26926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:47.243527 26923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:47.243925 26918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:47.244174 26918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:47.245325 26918 hybrid_clock.cc:648] HybridClock initialized: now 1790598887245316 us; error 32 us; skew 500 ppm
I20260928 12:34:47.251757 26918 webserver.cc:460] Webserver started at http://127.8.13.1:35593/ using document root <none> and password file <none>
I20260928 12:34:47.252035 26918 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:47.252089 26918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:47.252223 26918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:47.253355 26918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "06bb47effb9740c1affe2726aa538bcd"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "edfeb335cfb8f18ed1baa71d7f609ef1"
server_key_iv: "5a9620629e1192412426a7913afb2a67"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.253723 26918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "06bb47effb9740c1affe2726aa538bcd"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "edfeb335cfb8f18ed1baa71d7f609ef1"
server_key_iv: "5a9620629e1192412426a7913afb2a67"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.255101 26918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:34:47.255878 26932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:47.256093 26918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:34:47.256174 26918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "06bb47effb9740c1affe2726aa538bcd"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "edfeb335cfb8f18ed1baa71d7f609ef1"
server_key_iv: "5a9620629e1192412426a7913afb2a67"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.256249 26918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:47.266624 26918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:47.266952 26918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:47.267083 26918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:47.267306 26918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:47.267650 26918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:47.267688 26918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:47.267719 26918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:47.267743 26918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:47.273370 26918 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:37351
I20260928 12:34:47.273416 27045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:37351 every 8 connection(s)
I20260928 12:34:47.273774 26918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:34:47.279232 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 26918
I20260928 12:34:47.279284 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.274557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:33593 (local address 127.8.13.62:40601)
0928 12:34:47.274641 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:47.274643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:47.274845 (+ 202us) server_negotiation.cc:407] Connection header received
0928 12:34:47.275428 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:47.275430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:47.275447 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:47.275500 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:47.276175 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.276688 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.277790 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.278041 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.278102 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.279002 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:47.279004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:47.279004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:47.279010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:47.279027 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:47.279032 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:47.279076 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:47.279154 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:34:47.279185 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 12:34:47.279351 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:34:47.279484 27048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.274532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40601 (local address 127.8.13.1:33593)
0928 12:34:47.274722 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:47.274759 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:47.275292 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:47.275573 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:47.275576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:47.275586 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:47.276099 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.276103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.276764 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.276765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.277685 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.277688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.278125 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.278126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.278272 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.278854 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:34:47.278854 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:47.278886 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:34:47.278913 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:47.279083 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:47.279089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:47.279092 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:47.279154 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:34:47.279274 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260928 12:34:47.279588 8244 external_mini_cluster.cc:1428] Setting key c7d4991fe592dba4fb908d37554ab4db
I20260928 12:34:47.280084 27046 heartbeater.cc:347] Connected to a master server at 127.8.13.62:40601
I20260928 12:34:47.280198 27046 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:47.280413 27046 heartbeater.cc:511] Master 127.8.13.62:40601 requested a full tablet report, sending...
I20260928 12:34:47.280525 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40601
--builtin_ntp_servers=127.8.13.20:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 12:34:47.281044 26858 ts_manager.cc:194] Registered new tserver with Master: 06bb47effb9740c1affe2726aa538bcd (127.8.13.1:37351)
I20260928 12:34:47.281651 26858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.1:33593
W20260928 12:34:47.360945 27049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:47.361173 27049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:47.361197 27049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:47.362488 27049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:47.362538 27049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:47.362571 27049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:34:47.364027 27049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:47.364280 27049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:47.364583 27049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:47.369628 27057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:47.369628 27054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:47.369649 27055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:47.369874 27049 server_base.cc:1034] running on GCE node
I20260928 12:34:47.370101 27049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:47.370332 27049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:47.371465 27049 hybrid_clock.cc:648] HybridClock initialized: now 1790598887371446 us; error 35 us; skew 500 ppm
I20260928 12:34:47.372781 27049 webserver.cc:460] Webserver started at http://127.8.13.2:38451/ using document root <none> and password file <none>
I20260928 12:34:47.373027 27049 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:47.373075 27049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:47.373202 27049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:47.374341 27049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "c361c71ffe7e40b384bc56e327dd0d96"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "ec97f93cf7c8fd1e4924fac2626c5478"
server_key_iv: "cd9ff9772459e030a11321d83d9089fc"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.374683 27049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "c361c71ffe7e40b384bc56e327dd0d96"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "ec97f93cf7c8fd1e4924fac2626c5478"
server_key_iv: "cd9ff9772459e030a11321d83d9089fc"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.375945 27049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:34:47.376601 27063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:47.376751 27049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:47.376837 27049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "c361c71ffe7e40b384bc56e327dd0d96"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "ec97f93cf7c8fd1e4924fac2626c5478"
server_key_iv: "cd9ff9772459e030a11321d83d9089fc"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.376893 27049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:47.409662 27049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:47.410006 27049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:47.410115 27049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:47.410300 27049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:47.410642 27049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:47.410684 27049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:47.410708 27049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:47.410722 27049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:47.415867 27049 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:35037
I20260928 12:34:47.415912 27176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:35037 every 8 connection(s)
I20260928 12:34:47.416221 27049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:34:47.421005 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.416875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:36469 (local address 127.8.13.62:40601)
0928 12:34:47.416929 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:47.416930 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:47.417140 (+ 210us) server_negotiation.cc:407] Connection header received
0928 12:34:47.417607 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:47.417608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:47.417619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:47.417686 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:47.418345 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.418826 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.419821 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.420016 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.420080 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.420799 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:47.420800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:47.420800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:47.420805 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:47.420821 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:47.420825 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:47.420869 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:47.420921 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 12:34:47.420946 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:34:47.421207 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.416853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40601 (local address 127.8.13.2:36469)
0928 12:34:47.417032 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:47.417069 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:47.417522 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:47.417740 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:47.417743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:47.417751 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:47.418275 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.418280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.418915 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.418916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.419708 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.419711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.420090 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.420091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.420196 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.420672 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:34:47.420673 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:47.420699 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:34:47.420721 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:47.420873 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:47.420875 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:47.420878 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:47.420923 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:34:47.421006 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260928 12:34:47.421643 27177 heartbeater.cc:347] Connected to a master server at 127.8.13.62:40601
I20260928 12:34:47.421722 27177 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:47.421888 27177 heartbeater.cc:511] Master 127.8.13.62:40601 requested a full tablet report, sending...
I20260928 12:34:47.422215 26858 ts_manager.cc:194] Registered new tserver with Master: c361c71ffe7e40b384bc56e327dd0d96 (127.8.13.2:35037)
I20260928 12:34:47.422554 26858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.2:36469
I20260928 12:34:47.425619 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 27049
I20260928 12:34:47.425709 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:34:47.425931 8244 external_mini_cluster.cc:1428] Setting key c6bdd316dde2d734630ed0e848467e52
I20260928 12:34:47.426745 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40601
--builtin_ntp_servers=127.8.13.20:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:34:47.506379 27180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:47.506626 27180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:47.506660 27180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:47.507896 27180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:47.507946 27180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:47.507982 27180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:34:47.509443 27180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:47.509711 27180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:47.510039 27180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:47.515033 27185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:47.515035 27186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:47.515044 27188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:47.515210 27180 server_base.cc:1034] running on GCE node
I20260928 12:34:47.515408 27180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:47.515633 27180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:47.516763 27180 hybrid_clock.cc:648] HybridClock initialized: now 1790598887516747 us; error 34 us; skew 500 ppm
I20260928 12:34:47.518024 27180 webserver.cc:460] Webserver started at http://127.8.13.3:44473/ using document root <none> and password file <none>
I20260928 12:34:47.518250 27180 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:47.518294 27180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:47.518419 27180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:47.519445 27180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "f3bcffda4ca9417c92e4428e076ad916"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "03789db83bf7802e8fab931e076e079a"
server_key_iv: "a97ea783e8fa3bab84d75d1dee9e5524"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.519762 27180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "f3bcffda4ca9417c92e4428e076ad916"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "03789db83bf7802e8fab931e076e079a"
server_key_iv: "a97ea783e8fa3bab84d75d1dee9e5524"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.520920 27180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:34:47.521601 27194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:47.521749 27180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:47.521852 27180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "f3bcffda4ca9417c92e4428e076ad916"
format_stamp: "Formatted at 2026-09-28 12:34:47 on dist-test-slave-g31s"
server_key: "03789db83bf7802e8fab931e076e079a"
server_key_iv: "a97ea783e8fa3bab84d75d1dee9e5524"
server_key_version: "encryptionkey@0"
I20260928 12:34:47.521916 27180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:47.533209 27180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:47.533468 27180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:47.533564 27180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:47.533758 27180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:47.534097 27180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:47.534132 27180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:47.534161 27180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:47.534186 27180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:47.539587 27180 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:33331
I20260928 12:34:47.539647 27307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:33331 every 8 connection(s)
I20260928 12:34:47.539984 27180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:34:47.541026 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 27180
I20260928 12:34:47.541127 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:34:47.541311 8244 external_mini_cluster.cc:1428] Setting key 2952b79211ddaa04a581b9342d442db0
I20260928 12:34:47.545217 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.540696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:58905 (local address 127.8.13.62:40601)
0928 12:34:47.540750 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:47.540752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:47.540999 (+ 247us) server_negotiation.cc:407] Connection header received
0928 12:34:47.541659 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:47.541660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:47.541672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:47.541718 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:47.542475 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.542959 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.543933 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.544132 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.544187 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.544950 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:47.544951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:47.544951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:47.544957 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:47.544975 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:47.544979 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:47.545027 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:47.545106 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 12:34:47.545134 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:34:47.545408 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.540681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:40601 (local address 127.8.13.3:58905)
0928 12:34:47.540871 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:47.540908 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:47.541567 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:47.541819 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:47.541823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:47.541832 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:47.542392 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.542398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.543054 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.543054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.543851 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.543853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.544205 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.544206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.544346 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.544822 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:34:47.544823 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:47.544852 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:34:47.544876 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:47.545033 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:47.545038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:47.545041 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:47.545101 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:34:47.545205 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260928 12:34:47.545856 27308 heartbeater.cc:347] Connected to a master server at 127.8.13.62:40601
I20260928 12:34:47.545960 27308 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:47.546139 27308 heartbeater.cc:511] Master 127.8.13.62:40601 requested a full tablet report, sending...
I20260928 12:34:47.546482 26858 ts_manager.cc:194] Registered new tserver with Master: f3bcffda4ca9417c92e4428e076ad916 (127.8.13.3:33331)
I20260928 12:34:47.546826 26858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.3:58905
I20260928 12:34:47.553117 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:34:47.557408 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.554522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.8.13.62:40601)
0928 12:34:47.554588 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:47.554590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:47.554700 (+ 110us) server_negotiation.cc:407] Connection header received
0928 12:34:47.554747 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:47.554747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:47.554757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:47.554805 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:47.554947 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.555438 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.556195 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.556370 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.556413 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.557079 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:47.557081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:47.557081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:47.557086 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:47.557102 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:47.557106 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:47.557188 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:47.557297 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 12:34:47.557310 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:34:47.559170 26858 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:34:47.560781 26858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:34:47.567162 27319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.563096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34050 (local address 127.8.13.2:35037)
0928 12:34:47.563289 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:47.563293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:47.563304 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:34:47.563363 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:47.563365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:47.563394 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:47.563499 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:47.563699 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.564613 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.566017 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.566747 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.566827 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.566926 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:47.567029 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 12:34:47.567072 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260928 12:34:47.567159 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.562996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:35037 (local address 127.0.0.1:34050)
0928 12:34:47.563174 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:47.563215 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:47.563282 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:47.563524 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:47.563528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:47.563530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:47.563607 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.563609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.564684 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.564685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.565914 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.565916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.566837 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.566838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.566927 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.566942 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 12:34:47.567009 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 12:34:47.567068 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260928 12:34:47.567415 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.563439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.8.13.3:33331)
0928 12:34:47.564016 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:47.564023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:47.564033 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:34:47.564072 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:47.564074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:47.564089 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:47.564165 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:47.564288 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.565255 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.566348 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.566853 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.566922 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.567010 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:47.567216 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 12:34:47.567323 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":143,"threads_started":1}
I20260928 12:34:47.567480 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.563093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.8.13.1:37351)
0928 12:34:47.563414 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:47.563418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:47.563427 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:34:47.563463 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:47.563465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:47.563479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:47.563542 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:47.563853 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.564961 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.566196 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.566894 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.566950 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.567027 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:47.567354 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 12:34:47.567400 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":80,"threads_started":1}
I20260928 12:34:47.567279 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.563015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:37351 (local address 127.0.0.1:42544)
0928 12:34:47.563279 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:47.563290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:47.563339 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:47.563614 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:47.563615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:47.563617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:47.563668 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.563670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.565047 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.565048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.566113 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.566115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.567060 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.567061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.567119 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.567123 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:34:47.567170 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:34:47.567212 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260928 12:34:47.567374 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.563217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:33331 (local address 127.0.0.1:58202)
0928 12:34:47.563668 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:47.563677 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:47.563717 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:47.564179 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:47.564179 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:47.564181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:47.564219 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.564221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.565325 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.565326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.566270 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.566272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.567045 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.567046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.567097 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.567101 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:34:47.567142 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 12:34:47.567176 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260928 12:34:47.568411 27242 tablet_service.cc:1467] Processing CreateTablet for tablet aedab4d055e24d9586d5345315cbaadc (DEFAULT_TABLE table=test-table [id=a9aa3f4a7db847c8923a6e019453a808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:47.568413 27111 tablet_service.cc:1467] Processing CreateTablet for tablet aedab4d055e24d9586d5345315cbaadc (DEFAULT_TABLE table=test-table [id=a9aa3f4a7db847c8923a6e019453a808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:47.568799 27111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aedab4d055e24d9586d5345315cbaadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:47.568799 27242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aedab4d055e24d9586d5345315cbaadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:47.570225 26980 tablet_service.cc:1467] Processing CreateTablet for tablet aedab4d055e24d9586d5345315cbaadc (DEFAULT_TABLE table=test-table [id=a9aa3f4a7db847c8923a6e019453a808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:47.570537 26980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aedab4d055e24d9586d5345315cbaadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:47.572420 27322 tablet_bootstrap.cc:492] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96: Bootstrap starting.
I20260928 12:34:47.572613 27323 tablet_bootstrap.cc:492] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916: Bootstrap starting.
I20260928 12:34:47.573305 27323 tablet_bootstrap.cc:654] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:47.573390 27324 tablet_bootstrap.cc:492] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd: Bootstrap starting.
I20260928 12:34:47.573643 27323 log.cc:824] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:47.574023 27324 tablet_bootstrap.cc:654] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:47.574322 27324 log.cc:824] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:47.574451 27323 tablet_bootstrap.cc:492] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916: No bootstrap required, opened a new log
I20260928 12:34:47.574467 27322 tablet_bootstrap.cc:654] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:47.574519 27323 ts_tablet_manager.cc:1397] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:34:47.574743 27322 log.cc:824] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:47.575063 27324 tablet_bootstrap.cc:492] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd: No bootstrap required, opened a new log
I20260928 12:34:47.575125 27324 ts_tablet_manager.cc:1397] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:34:47.575536 27322 tablet_bootstrap.cc:492] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96: No bootstrap required, opened a new log
I20260928 12:34:47.575616 27322 ts_tablet_manager.cc:1397] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 12:34:47.576633 27323 raft_consensus.cc:348] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c361c71ffe7e40b384bc56e327dd0d96" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35037 } } peers { permanent_uuid: "06bb47effb9740c1affe2726aa538bcd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 37351 } } peers { permanent_uuid: "f3bcffda4ca9417c92e4428e076ad916" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33331 } }
I20260928 12:34:47.576797 27323 raft_consensus.cc:374] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:47.576836 27323 raft_consensus.cc:729] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bcffda4ca9417c92e4428e076ad916, State: Initialized, Role: FOLLOWER
I20260928 12:34:47.576980 27323 consensus_queue.cc:260] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c361c71ffe7e40b384bc56e327dd0d96" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35037 } } peers { permanent_uuid: "06bb47effb9740c1affe2726aa538bcd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 37351 } } peers { permanent_uuid: "f3bcffda4ca9417c92e4428e076ad916" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33331 } }
I20260928 12:34:47.577296 27323 ts_tablet_manager.cc:1428] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:34:47.577342 27308 heartbeater.cc:503] Master 127.8.13.62:40601 was elected leader, sending a full tablet report...
I20260928 12:34:47.577768 27324 raft_consensus.cc:348] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c361c71ffe7e40b384bc56e327dd0d96" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35037 } } peers { permanent_uuid: "06bb47effb9740c1affe2726aa538bcd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 37351 } } peers { permanent_uuid: "f3bcffda4ca9417c92e4428e076ad916" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33331 } }
I20260928 12:34:47.577980 27324 raft_consensus.cc:374] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:47.578017 27324 raft_consensus.cc:729] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06bb47effb9740c1affe2726aa538bcd, State: Initialized, Role: FOLLOWER
I20260928 12:34:47.578344 27324 consensus_queue.cc:260] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c361c71ffe7e40b384bc56e327dd0d96" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35037 } } peers { permanent_uuid: "06bb47effb9740c1affe2726aa538bcd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 37351 } } peers { permanent_uuid: "f3bcffda4ca9417c92e4428e076ad916" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33331 } }
I20260928 12:34:47.578491 27322 raft_consensus.cc:348] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c361c71ffe7e40b384bc56e327dd0d96" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35037 } } peers { permanent_uuid: "06bb47effb9740c1affe2726aa538bcd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 37351 } } peers { permanent_uuid: "f3bcffda4ca9417c92e4428e076ad916" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33331 } }
I20260928 12:34:47.578615 27324 ts_tablet_manager.cc:1428] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 12:34:47.578644 27322 raft_consensus.cc:374] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:47.578656 27046 heartbeater.cc:503] Master 127.8.13.62:40601 was elected leader, sending a full tablet report...
I20260928 12:34:47.578682 27322 raft_consensus.cc:729] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c361c71ffe7e40b384bc56e327dd0d96, State: Initialized, Role: FOLLOWER
I20260928 12:34:47.578788 27322 consensus_queue.cc:260] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c361c71ffe7e40b384bc56e327dd0d96" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35037 } } peers { permanent_uuid: "06bb47effb9740c1affe2726aa538bcd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 37351 } } peers { permanent_uuid: "f3bcffda4ca9417c92e4428e076ad916" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33331 } }
I20260928 12:34:47.579021 27322 ts_tablet_manager.cc:1428] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:34:47.579149 27177 heartbeater.cc:503] Master 127.8.13.62:40601 was elected leader, sending a full tablet report...
I20260928 12:34:47.604174 27330 raft_consensus.cc:482] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:34:47.604288 27330 raft_consensus.cc:504] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c361c71ffe7e40b384bc56e327dd0d96" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35037 } } peers { permanent_uuid: "06bb47effb9740c1affe2726aa538bcd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 37351 } } peers { permanent_uuid: "f3bcffda4ca9417c92e4428e076ad916" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33331 } }
I20260928 12:34:47.604573 27330 leader_election.cc:288] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06bb47effb9740c1affe2726aa538bcd (127.8.13.1:37351), f3bcffda4ca9417c92e4428e076ad916 (127.8.13.3:33331)
I20260928 12:34:47.608290 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.604778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:38859 (local address 127.8.13.1:37351)
0928 12:34:47.604837 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:47.604839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:47.604848 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:34:47.604904 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:47.604905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:47.604916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:47.604963 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:47.605134 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.605934 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.607377 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.607939 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.608022 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.608034 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:47.608182 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 12:34:47.608220 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:34:47.608290 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.604891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:34491 (local address 127.8.13.3:33331)
0928 12:34:47.604944 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:47.604946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:47.605040 (+ 94us) server_negotiation.cc:407] Connection header received
0928 12:34:47.605100 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:47.605100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:47.605109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:47.605160 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:47.605266 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.605995 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.607377 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.607970 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.608029 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.608037 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:47.608178 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 12:34:47.608214 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:34:47.608309 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.604719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:37351 (local address 127.8.13.2:38859)
0928 12:34:47.604772 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:47.604785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:47.604839 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:47.604991 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:47.605001 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:47.605003 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:47.605072 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.605073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.606022 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.606023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.607271 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.607273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.608035 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.608035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.608110 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.608113 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:47.608167 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:34:47.608214 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":55}
I20260928 12:34:47.608314 27331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.604813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:33331 (local address 127.8.13.2:34491)
0928 12:34:47.604975 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:47.604985 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:47.605040 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:47.605167 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:47.605168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:47.605169 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:47.605217 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.605218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.606051 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.606052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.607280 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:47.607282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.608034 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.608035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:47.608110 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.608113 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:47.608167 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:34:47.608209 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260928 12:34:47.608687 27000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedab4d055e24d9586d5345315cbaadc" candidate_uuid: "c361c71ffe7e40b384bc56e327dd0d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06bb47effb9740c1affe2726aa538bcd" is_pre_election: true
I20260928 12:34:47.608847 27000 raft_consensus.cc:2454] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c361c71ffe7e40b384bc56e327dd0d96 in term 0.
I20260928 12:34:47.608847 27262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedab4d055e24d9586d5345315cbaadc" candidate_uuid: "c361c71ffe7e40b384bc56e327dd0d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bcffda4ca9417c92e4428e076ad916" is_pre_election: true
I20260928 12:34:47.608989 27262 raft_consensus.cc:2454] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c361c71ffe7e40b384bc56e327dd0d96 in term 0.
I20260928 12:34:47.609097 27066 leader_election.cc:302] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06bb47effb9740c1affe2726aa538bcd, c361c71ffe7e40b384bc56e327dd0d96; no voters:
I20260928 12:34:47.609220 27330 raft_consensus.cc:2781] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:34:47.609294 27330 raft_consensus.cc:482] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:34:47.609318 27330 raft_consensus.cc:3037] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:47.610074 27330 raft_consensus.cc:504] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c361c71ffe7e40b384bc56e327dd0d96" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35037 } } peers { permanent_uuid: "06bb47effb9740c1affe2726aa538bcd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 37351 } } peers { permanent_uuid: "f3bcffda4ca9417c92e4428e076ad916" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33331 } }
I20260928 12:34:47.610193 27330 leader_election.cc:288] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [CANDIDATE]: Term 1 election: Requested vote from peers 06bb47effb9740c1affe2726aa538bcd (127.8.13.1:37351), f3bcffda4ca9417c92e4428e076ad916 (127.8.13.3:33331)
I20260928 12:34:47.610342 27000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedab4d055e24d9586d5345315cbaadc" candidate_uuid: "c361c71ffe7e40b384bc56e327dd0d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06bb47effb9740c1affe2726aa538bcd"
I20260928 12:34:47.610360 27262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedab4d055e24d9586d5345315cbaadc" candidate_uuid: "c361c71ffe7e40b384bc56e327dd0d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bcffda4ca9417c92e4428e076ad916"
I20260928 12:34:47.610420 27262 raft_consensus.cc:3037] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:47.610425 27000 raft_consensus.cc:3037] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:47.611088 27262 raft_consensus.cc:2454] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c361c71ffe7e40b384bc56e327dd0d96 in term 1.
I20260928 12:34:47.611181 27000 raft_consensus.cc:2454] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c361c71ffe7e40b384bc56e327dd0d96 in term 1.
I20260928 12:34:47.611254 27066 leader_election.cc:302] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c361c71ffe7e40b384bc56e327dd0d96, f3bcffda4ca9417c92e4428e076ad916; no voters:
I20260928 12:34:47.611361 27330 raft_consensus.cc:2781] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:47.611474 27330 raft_consensus.cc:686] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [term 1 LEADER]: Becoming Leader. State: Replica: c361c71ffe7e40b384bc56e327dd0d96, State: Running, Role: LEADER
I20260928 12:34:47.611569 27330 consensus_queue.cc:237] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c361c71ffe7e40b384bc56e327dd0d96" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35037 } } peers { permanent_uuid: "06bb47effb9740c1affe2726aa538bcd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 37351 } } peers { permanent_uuid: "f3bcffda4ca9417c92e4428e076ad916" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33331 } }
I20260928 12:34:47.612442 26857 catalog_manager.cc:5495] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 reported cstate change: term changed from 0 to 1, leader changed from <none> to c361c71ffe7e40b384bc56e327dd0d96 (127.8.13.2). New cstate: current_term: 1 leader_uuid: "c361c71ffe7e40b384bc56e327dd0d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c361c71ffe7e40b384bc56e327dd0d96" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 35037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06bb47effb9740c1affe2726aa538bcd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 37351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3bcffda4ca9417c92e4428e076ad916" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33331 } health_report { overall_health: UNKNOWN } } }
I20260928 12:34:47.617436 27319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.615444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.8.13.2:35037)
0928 12:34:47.615482 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:47.615483 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:47.615490 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:34:47.615562 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:47.615562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:47.615578 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:47.615636 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:34:47.615782 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.616226 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.616866 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:47.617034 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:47.617075 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:47.617160 (+ 85us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:34:47.617237 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:34:47.617298 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:47.617343 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 12:34:47.617351 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:34:47.619031 27000 raft_consensus.cc:1261] T aedab4d055e24d9586d5345315cbaadc P 06bb47effb9740c1affe2726aa538bcd [term 1 FOLLOWER]: Refusing update from remote peer c361c71ffe7e40b384bc56e327dd0d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:47.619047 27262 raft_consensus.cc:1261] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916 [term 1 FOLLOWER]: Refusing update from remote peer c361c71ffe7e40b384bc56e327dd0d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:47.619352 27332 consensus_queue.cc:1035] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3bcffda4ca9417c92e4428e076ad916" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:47.619474 27330 consensus_queue.cc:1035] T aedab4d055e24d9586d5345315cbaadc P c361c71ffe7e40b384bc56e327dd0d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06bb47effb9740c1affe2726aa538bcd" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 37351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:47.621670 27335 mvcc.cc:205] Tried to move back new op lower bound from 7334293043685380096 to 7334293043657576448. Current Snapshot: MvccSnapshot[applied={T|T < 7334293043685380096}]
I20260928 12:34:47.621830 27337 mvcc.cc:205] Tried to move back new op lower bound from 7334293043685380096 to 7334293043657576448. Current Snapshot: MvccSnapshot[applied={T|T < 7334293043685380096}]
I20260928 12:34:47.624087 26856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:47.624169 26856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:47.624823 26856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a93a6a13d5a84c50b4200955c8591d27: Sending DeleteTablet for 3 replicas of tablet aedab4d055e24d9586d5345315cbaadc
I20260928 12:34:47.625137 27111 tablet_service.cc:1514] Processing DeleteTablet for tablet aedab4d055e24d9586d5345315cbaadc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:47 UTC) from {username='slave'} at 127.0.0.1:34050
I20260928 12:34:47.625178 27242 tablet_service.cc:1514] Processing DeleteTablet for tablet aedab4d055e24d9586d5345315cbaadc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:47 UTC) from {username='slave'} at 127.0.0.1:58202
I20260928 12:34:47.625204 26980 tablet_service.cc:1514] Processing DeleteTablet for tablet aedab4d055e24d9586d5345315cbaadc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:47 UTC) from {username='slave'} at 127.0.0.1:42544
I20260928 12:34:47.625376 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 26918
W20260928 12:34:47.629865 26844 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:37351 (error 108)
I20260928 12:34:47.630098 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 27049
I20260928 12:34:47.630249 27346 tablet_replica.cc:330] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916: stopping tablet replica
I20260928 12:34:47.630400 27346 raft_consensus.cc:2229] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:34:47.630640 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.630189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:37351 (local address 127.0.0.1:42548)
0928 12:34:47.630568 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:47.630609 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:37351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359}
I20260928 12:34:47.630654 27346 raft_consensus.cc:2258] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 12:34:47.630980 26844 catalog_manager.cc:4584] TS 06bb47effb9740c1affe2726aa538bcd (127.8.13.1:37351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aedab4d055e24d9586d5345315cbaadc: Network error: Client connection negotiation failed: client connection to 127.8.13.1:37351: connect: Connection refused (error 111)
I20260928 12:34:47.631311 27346 ts_tablet_manager.cc:1905] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:47.632794 27346 ts_tablet_manager.cc:1918] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:47.632864 27346 log.cc:1196] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/aedab4d055e24d9586d5345315cbaadc
I20260928 12:34:47.633085 27346 ts_tablet_manager.cc:1939] T aedab4d055e24d9586d5345315cbaadc P f3bcffda4ca9417c92e4428e076ad916: Deleting consensus metadata
I20260928 12:34:47.633476 26844 catalog_manager.cc:4841] TS f3bcffda4ca9417c92e4428e076ad916 (127.8.13.3:33331): tablet aedab4d055e24d9586d5345315cbaadc (table test-table [id=a9aa3f4a7db847c8923a6e019453a808]) successfully deleted
I20260928 12:34:47.635156 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 27180
I20260928 12:34:47.635272 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:47.635145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:35037 (local address 127.0.0.1:34064)
0928 12:34:47.635202 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:47.635232 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.2:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:34:47.640338 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 26827
2026-09-28T12:34:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (687 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:47 dist-test-slave-g31s krb5kdc[27351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:47 dist-test-slave-g31s krb5kdc[27351](info): set up 2 sockets
Sep 28 12:34:47 dist-test-slave-g31s krb5kdc[27351](info): commencing operation
krb5kdc: starting...
W20260928 12:34:48.585474 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:34:48 dist-test-slave-g31s krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:34:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:34:48.702775 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33633
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:33633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:48.782743 27364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:48.782963 27364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:48.782995 27364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:48.784193 27364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:48.784236 27364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:48.784255 27364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:48.784271 27364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:48.784287 27364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:48.785844 27364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:33633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:48.786091 27364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:48.786301 27364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:48.790938 27372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:48.791057 27364 server_base.cc:1034] running on GCE node
W20260928 12:34:48.790938 27370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:48.790943 27369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:48.791332 27364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:48.791613 27364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:48.792761 27364 hybrid_clock.cc:648] HybridClock initialized: now 1790598888792753 us; error 41 us; skew 500 ppm
Sep 28 12:34:48 dist-test-slave-g31s krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598888, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:48.795253 27364 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:34:48.795682 27364 webserver.cc:460] Webserver started at http://127.8.13.62:34313/ using document root <none> and password file <none>
I20260928 12:34:48.795895 27364 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:48.795939 27364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:48.796033 27364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:48.797147 27364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "9dcb4d01c34e4c62ad02c989ef6db6ef"
format_stamp: "Formatted at 2026-09-28 12:34:48 on dist-test-slave-g31s"
server_key: "adb4f826e0d2f1833550b0652acd23b1"
server_key_iv: "bc5546ddced83d2919379a68048243db"
server_key_version: "encryptionkey@0"
I20260928 12:34:48.797475 27364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "9dcb4d01c34e4c62ad02c989ef6db6ef"
format_stamp: "Formatted at 2026-09-28 12:34:48 on dist-test-slave-g31s"
server_key: "adb4f826e0d2f1833550b0652acd23b1"
server_key_iv: "bc5546ddced83d2919379a68048243db"
server_key_version: "encryptionkey@0"
I20260928 12:34:48.798815 27364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:34:48.799535 27379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:48.799711 27364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:34:48.799801 27364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "9dcb4d01c34e4c62ad02c989ef6db6ef"
format_stamp: "Formatted at 2026-09-28 12:34:48 on dist-test-slave-g31s"
server_key: "adb4f826e0d2f1833550b0652acd23b1"
server_key_iv: "bc5546ddced83d2919379a68048243db"
server_key_version: "encryptionkey@0"
I20260928 12:34:48.799865 27364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:48.817870 27364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:48.820031 27364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:48.820165 27364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:48.823941 27364 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:33633
I20260928 12:34:48.823979 27431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:33633 every 8 connection(s)
I20260928 12:34:48.824286 27364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:48.825086 27432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:48.827309 27432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef: Bootstrap starting.
I20260928 12:34:48.827454 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 27364
I20260928 12:34:48.827553 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:48.827718 8244 external_mini_cluster.cc:1428] Setting key 879ed20ccaf8dba91f7a9a4f00e7099b
I20260928 12:34:48.827911 27432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:48.828228 27432 log.cc:824] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:48.828950 27432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef: No bootstrap required, opened a new log
I20260928 12:34:48.831178 27432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb4d01c34e4c62ad02c989ef6db6ef" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33633 } }
I20260928 12:34:48.831341 27432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:48.831377 27432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcb4d01c34e4c62ad02c989ef6db6ef, State: Initialized, Role: FOLLOWER
I20260928 12:34:48.831496 27432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb4d01c34e4c62ad02c989ef6db6ef" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33633 } }
I20260928 12:34:48.831571 27432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:48.831601 27432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:48.831646 27432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:48.832244 27432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb4d01c34e4c62ad02c989ef6db6ef" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33633 } }
I20260928 12:34:48.832353 27432 leader_election.cc:302] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dcb4d01c34e4c62ad02c989ef6db6ef; no voters:
I20260928 12:34:48.832517 27432 leader_election.cc:288] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:48.832554 27437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:48.832705 27437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [term 1 LEADER]: Becoming Leader. State: Replica: 9dcb4d01c34e4c62ad02c989ef6db6ef, State: Running, Role: LEADER
I20260928 12:34:48.832777 27432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:48.832803 27437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb4d01c34e4c62ad02c989ef6db6ef" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33633 } }
I20260928 12:34:48.833451 27437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dcb4d01c34e4c62ad02c989ef6db6ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb4d01c34e4c62ad02c989ef6db6ef" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33633 } } }
I20260928 12:34:48.833529 27437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:48.833636 27437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dcb4d01c34e4c62ad02c989ef6db6ef. Latest consensus state: current_term: 1 leader_uuid: "9dcb4d01c34e4c62ad02c989ef6db6ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb4d01c34e4c62ad02c989ef6db6ef" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 33633 } } }
I20260928 12:34:48.833690 27437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:48.833841 27446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:48.834447 27446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 12:34:48 dist-test-slave-g31s krb5kdc[27351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:48.836016 27446 catalog_manager.cc:1347] Generated new cluster ID: 33b869aa1ff84e12bd3d6e2335d9ce0d
I20260928 12:34:48.836071 27446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:48.839323 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:48.828184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.8.13.62:33633)
0928 12:34:48.828345 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:48.828354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:48.828392 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:34:48.828924 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:48.828932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:48.828993 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:48.829155 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:48.829710 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:48.831119 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:48.831944 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:48.832569 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:48.832681 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:48.835619 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:48.835624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:48.835625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:48.835637 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:48.837614 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:48.838082 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:48.838085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:48.838086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:48.838191 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:48.838389 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:48.838389 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:48.838389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:48.838680 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:48.838833 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:48.838970 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 12:34:48.839109 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260928 12:34:48.847431 27446 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:48.847987 27446 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:48.853248 27446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef: Generated new TSK 0
I20260928 12:34:48.853463 27446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
Entry for principal kudu/127.8.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab.
I20260928 12:34:48.878647 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33633
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:48.963959 27458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:48.964200 27458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:48.964233 27458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:48.965545 27458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:48.965600 27458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:48.965631 27458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:34:48.967365 27458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.1.keytab
--principal=kudu/127.8.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:48.967598 27458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:48.967796 27458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:48.973060 27464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:48.973032 27463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:48.973073 27458 server_base.cc:1034] running on GCE node
W20260928 12:34:48.973032 27466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:48.973510 27458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:48.973754 27458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:48.974943 27458 hybrid_clock.cc:648] HybridClock initialized: now 1790598888974927 us; error 30 us; skew 500 ppm
Sep 28 12:34:48 dist-test-slave-g31s krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598888, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:48.977451 27458 init.cc:377] Logged in from keytab as kudu/127.8.13.1@KRBTEST.COM (short username kudu)
I20260928 12:34:48.977978 27458 webserver.cc:460] Webserver started at http://127.8.13.1:35241/ using document root <none> and password file <none>
I20260928 12:34:48.978193 27458 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:48.978245 27458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:48.978384 27458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:48.979478 27458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "772a963153eb411eb66479e25b769004"
format_stamp: "Formatted at 2026-09-28 12:34:48 on dist-test-slave-g31s"
server_key: "eb71be2ef0ba19e7f6d149b0324cd83e"
server_key_iv: "a60410ebe52694b039e16c471172d137"
server_key_version: "encryptionkey@0"
I20260928 12:34:48.979831 27458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "772a963153eb411eb66479e25b769004"
format_stamp: "Formatted at 2026-09-28 12:34:48 on dist-test-slave-g31s"
server_key: "eb71be2ef0ba19e7f6d149b0324cd83e"
server_key_iv: "a60410ebe52694b039e16c471172d137"
server_key_version: "encryptionkey@0"
I20260928 12:34:48.981091 27458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:34:48.981757 27473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:48.981920 27458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:48.982004 27458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "772a963153eb411eb66479e25b769004"
format_stamp: "Formatted at 2026-09-28 12:34:48 on dist-test-slave-g31s"
server_key: "eb71be2ef0ba19e7f6d149b0324cd83e"
server_key_iv: "a60410ebe52694b039e16c471172d137"
server_key_version: "encryptionkey@0"
I20260928 12:34:48.982074 27458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:48.998442 27458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:49.000324 27458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:49.000504 27458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:49.000747 27458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:49.001062 27458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:49.001096 27458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.001128 27458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:49.001152 27458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.006889 27458 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:43769
I20260928 12:34:49.006928 27586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:43769 every 8 connection(s)
I20260928 12:34:49.007233 27458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:34:49 dist-test-slave-g31s krb5kdc[27351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598888, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.1@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:49.013826 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 27458
I20260928 12:34:49.013927 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:34:49.014077 8244 external_mini_cluster.cc:1428] Setting key c15b9404da9033cddcfb639a1866f214
I20260928 12:34:49.015841 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.007904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:45829 (local address 127.8.13.62:33633)
0928 12:34:49.007963 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.007965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.008144 (+ 179us) server_negotiation.cc:407] Connection header received
0928 12:34:49.008608 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.008609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.008617 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.008673 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:49.009584 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.010132 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.011082 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.011274 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.011315 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.013237 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:49.013239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:49.013239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:49.013246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:49.014538 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:49.014990 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:49.014991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:49.014991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:49.015051 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:49.015287 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:49.015288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:49.015288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:49.015381 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:49.015425 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.015700 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.015776 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:34:49.016065 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.007869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33633 (local address 127.8.13.1:45829)
0928 12:34:49.008015 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:49.008068 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:49.008535 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:49.008727 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:49.008730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:49.009306 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:49.009510 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.009514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.010209 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.010210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.011018 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.011020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.011327 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.011327 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.011418 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.011883 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:49.011891 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:49.013156 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:49.014597 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:49.014605 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:49.014607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:49.014927 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:49.015105 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:49.015106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:49.015106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:49.015234 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:49.015428 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:49.015433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:49.015583 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 12:34:49.015687 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 12:34:49.015848 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260928 12:34:49.016484 27587 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33633
I20260928 12:34:49.016569 27587 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:49.016736 27587 heartbeater.cc:511] Master 127.8.13.62:33633 requested a full tablet report, sending...
I20260928 12:34:49.017217 27396 ts_manager.cc:194] Registered new tserver with Master: 772a963153eb411eb66479e25b769004 (127.8.13.1:43769)
I20260928 12:34:49.017843 27396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.1@KRBTEST.COM'} at 127.8.13.1:45829
WARNING: no policy specified for kudu/127.8.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
Entry for principal kudu/127.8.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab.
I20260928 12:34:49.036384 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33633
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:49.117182 27592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:49.117399 27592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:49.117430 27592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:49.118652 27592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:49.118700 27592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:49.118736 27592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:34:49.120194 27592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.2.keytab
--principal=kudu/127.8.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:49.120427 27592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:49.120636 27592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:49.125269 27597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:49.125404 27598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:49.125300 27600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:49.125371 27592 server_base.cc:1034] running on GCE node
I20260928 12:34:49.125675 27592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:49.125927 27592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:49.127085 27592 hybrid_clock.cc:648] HybridClock initialized: now 1790598889127071 us; error 29 us; skew 500 ppm
Sep 28 12:34:49 dist-test-slave-g31s krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598889, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:49.129475 27592 init.cc:377] Logged in from keytab as kudu/127.8.13.2@KRBTEST.COM (short username kudu)
I20260928 12:34:49.129904 27592 webserver.cc:460] Webserver started at http://127.8.13.2:35457/ using document root <none> and password file <none>
I20260928 12:34:49.130106 27592 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:49.130148 27592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:49.130277 27592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:49.131345 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "942c5ce4907342c1b840288b774baec2"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "390eba449ed532d4f92d84466cb07cfa"
server_key_iv: "d58df7a7ff9da152bb8b32109b36b173"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.131654 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "942c5ce4907342c1b840288b774baec2"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "390eba449ed532d4f92d84466cb07cfa"
server_key_iv: "d58df7a7ff9da152bb8b32109b36b173"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.132920 27592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:34:49.133667 27607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.133846 27592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:49.133937 27592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "942c5ce4907342c1b840288b774baec2"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "390eba449ed532d4f92d84466cb07cfa"
server_key_iv: "d58df7a7ff9da152bb8b32109b36b173"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.134029 27592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:49.141358 27592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:49.143348 27592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:49.143488 27592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:49.143699 27592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:49.143990 27592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:49.144022 27592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.144052 27592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:49.144075 27592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.149466 27592 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:45881
I20260928 12:34:49.149523 27720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:45881 every 8 connection(s)
I20260928 12:34:49.149839 27592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:34:49.151014 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 27592
I20260928 12:34:49.151114 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:34:49.151304 8244 external_mini_cluster.cc:1428] Setting key 1324906eb4ff18fed307ae6c469a56d0
Sep 28 12:34:49 dist-test-slave-g31s krb5kdc[27351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598889, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.2@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.3@KRBTEST.COM; defaulting to no policy
I20260928 12:34:49.159641 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.150531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:56965 (local address 127.8.13.62:33633)
0928 12:34:49.150577 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.150579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.150771 (+ 192us) server_negotiation.cc:407] Connection header received
0928 12:34:49.151440 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.151442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.151456 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.151549 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:49.152709 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.153242 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.154305 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.154522 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.154578 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.156852 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:49.156854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:49.156855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:49.156863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:49.158239 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:49.158636 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:49.158638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:49.158638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:49.158693 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:49.158940 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:49.158941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:49.158941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:49.159074 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:49.159131 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.159410 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.159501 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:34:49.159852 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.150506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33633 (local address 127.8.13.2:56965)
0928 12:34:49.150654 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:49.150696 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:49.151304 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:49.151627 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:49.151632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:49.152333 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:49.152616 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.152622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.153346 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.153347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.154223 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.154225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.154593 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.154594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.154708 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.155192 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:49.155201 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:49.156751 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:49.158322 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:49.158331 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:49.158333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:49.158566 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:49.158748 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:49.158749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:49.158749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:49.158883 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:49.159130 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:49.159138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:49.159287 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 12:34:49.159378 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 12:34:49.159560 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260928 12:34:49.160316 27721 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33633
I20260928 12:34:49.160411 27721 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:49.160612 27721 heartbeater.cc:511] Master 127.8.13.62:33633 requested a full tablet report, sending...
I20260928 12:34:49.161067 27396 ts_manager.cc:194] Registered new tserver with Master: 942c5ce4907342c1b840288b774baec2 (127.8.13.2:45881)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.3@KRBTEST.COM" created.
I20260928 12:34:49.161460 27396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.2@KRBTEST.COM'} at 127.8.13.2:56965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
Entry for principal kudu/127.8.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab.
I20260928 12:34:49.175024 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33633
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:49.255733 27726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:49.255952 27726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:49.255976 27726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:49.257129 27726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:49.257170 27726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:49.257196 27726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:34:49.258790 27726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.3.keytab
--principal=kudu/127.8.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:49.258993 27726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:49.259188 27726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:49.264097 27734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:49.264127 27731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:49.264238 27732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:49.264470 27726 server_base.cc:1034] running on GCE node
I20260928 12:34:49.264704 27726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:49.264961 27726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:49.266106 27726 hybrid_clock.cc:648] HybridClock initialized: now 1790598889266089 us; error 32 us; skew 500 ppm
Sep 28 12:34:49 dist-test-slave-g31s krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598889, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:49.268620 27726 init.cc:377] Logged in from keytab as kudu/127.8.13.3@KRBTEST.COM (short username kudu)
I20260928 12:34:49.269096 27726 webserver.cc:460] Webserver started at http://127.8.13.3:33443/ using document root <none> and password file <none>
I20260928 12:34:49.269317 27726 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:49.269366 27726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:49.269490 27726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:49.270670 27726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "26d9677ca666793243ac175850c5d4f2"
server_key_iv: "0a2904b909fb0e8c71067708b9cbfada"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.271070 27726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "26d9677ca666793243ac175850c5d4f2"
server_key_iv: "0a2904b909fb0e8c71067708b9cbfada"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.272475 27726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:34:49.273281 27741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.273444 27726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:49.273528 27726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "26d9677ca666793243ac175850c5d4f2"
server_key_iv: "0a2904b909fb0e8c71067708b9cbfada"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.273667 27726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:49.292035 27726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:49.294234 27726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:49.294380 27726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:49.294608 27726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:49.294904 27726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:49.294936 27726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.294966 27726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:49.294983 27726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.300606 27726 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:33931
I20260928 12:34:49.300663 27854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:33931 every 8 connection(s)
I20260928 12:34:49.300953 27726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:34:49 dist-test-slave-g31s krb5kdc[27351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598889, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.3@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:34:49.309739 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.301624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:36859 (local address 127.8.13.62:33633)
0928 12:34:49.301671 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.301672 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.301901 (+ 229us) server_negotiation.cc:407] Connection header received
0928 12:34:49.302465 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.302466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.302479 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.302534 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:49.303515 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.304061 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.305030 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.305219 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.305273 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.307391 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:49.307393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:49.307393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:49.307400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:49.308552 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:49.308991 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:49.308991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:49.308992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:49.309037 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:49.309227 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:49.309227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:49.309227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:49.309329 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:49.309372 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.309611 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.309670 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:34:49.309975 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.301596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:33633 (local address 127.8.13.3:36859)
0928 12:34:49.301745 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:49.301788 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:49.302370 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:49.302614 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:49.302619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:49.303208 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:49.303430 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.303435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.304148 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.304149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.304944 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.304946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.305289 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.305290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.305417 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.305960 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:34:49.305968 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:49.307303 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:49.308662 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:49.308670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:49.308672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:49.308916 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:49.309084 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:34:49.309084 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:34:49.309085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:34:49.309184 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:34:49.309370 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:49.309372 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:49.309514 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 12:34:49.309594 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 12:34:49.309723 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260928 12:34:49.310114 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 27726
I20260928 12:34:49.310217 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:34:49.310384 8244 external_mini_cluster.cc:1428] Setting key 0cf34d568c4c531869863d727aeffed8
I20260928 12:34:49.310448 27855 heartbeater.cc:347] Connected to a master server at 127.8.13.62:33633
I20260928 12:34:49.310550 27855 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:49.310734 27855 heartbeater.cc:511] Master 127.8.13.62:33633 requested a full tablet report, sending...
I20260928 12:34:49.311157 27395 ts_manager.cc:194] Registered new tserver with Master: e99e9bb00c5e4c9ca587bc23bd7660ef (127.8.13.3:33931)
I20260928 12:34:49.311542 27395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.3@KRBTEST.COM'} at 127.8.13.3:36859
I20260928 12:34:49.322165 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:34:49.329118 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.323447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.8.13.62:33633)
0928 12:34:49.323498 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.323500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.323666 (+ 166us) server_negotiation.cc:407] Connection header received
0928 12:34:49.323693 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.323694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.323705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.323747 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:49.324272 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.324791 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.325485 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.325669 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.325723 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.326948 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:49.326949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:49.326950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:49.326956 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:49.328044 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:49.328344 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:49.328345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:49.328346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:49.328398 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:49.328530 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:49.328531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:49.328531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:49.328692 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:49.328784 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.328922 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.329026 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:34:49.330644 27395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:34:49.332159 27395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:34:49.338626 27865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.334393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:33931 (local address 127.0.0.1:50870)
0928 12:34:49.334837 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:49.334877 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:49.334947 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:49.335212 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:49.335215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:49.335217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:49.335300 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.335302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.336610 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.336611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.337583 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.337585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.338321 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.338322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.338400 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.338414 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:34:49.338480 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 12:34:49.338545 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":52,"threads_started":1}
I20260928 12:34:49.338637 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.334470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.8.13.3:33931)
0928 12:34:49.334804 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.334808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.334923 (+ 115us) server_negotiation.cc:407] Connection header received
0928 12:34:49.335016 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.335018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.335035 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.335116 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:49.335378 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.336493 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.337658 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.338245 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.338309 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.338369 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.338501 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.338545 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":98,"threads_started":1}
I20260928 12:34:49.339023 27863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.334398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:43769 (local address 127.0.0.1:42168)
0928 12:34:49.335111 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:49.335118 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:49.335155 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:49.335401 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:49.335402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:49.335404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:49.335434 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.335435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.336689 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.336690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.337829 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.337833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.338797 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.338798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.338853 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.338857 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:34:49.338906 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:34:49.338949 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260928 12:34:49.339071 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.334621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.8.13.1:43769)
0928 12:34:49.334768 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.334773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.335212 (+ 439us) server_negotiation.cc:407] Connection header received
0928 12:34:49.335255 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.335257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.335271 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.335331 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:49.335506 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.336594 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.337923 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.338647 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.338704 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.338760 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.338944 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.338992 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260928 12:34:49.339406 27789 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6022067a364828a756b4cee60ea6bb (DEFAULT_TABLE table=test-table [id=d474d21140a2478d903398278296711b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:49.339861 27789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6022067a364828a756b4cee60ea6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:49.340287 27864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.334396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:45881 (local address 127.0.0.1:33364)
0928 12:34:49.335020 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:49.335027 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:49.335060 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:49.335313 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:49.335314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:49.335317 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:49.335371 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.335373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.336367 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.336368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.338631 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.338635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.340064 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.340065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.340114 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.340117 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:49.340160 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:34:49.340206 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":68,"thread_start_us":60,"threads_started":1}
I20260928 12:34:49.340305 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.334644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.8.13.2:45881)
0928 12:34:49.334788 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.334792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.335098 (+ 306us) server_negotiation.cc:407] Connection header received
0928 12:34:49.335145 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.335147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.335163 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.335228 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:49.335455 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.336278 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.339277 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.339907 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.339986 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.340033 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.340192 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.340235 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260928 12:34:49.341127 27655 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6022067a364828a756b4cee60ea6bb (DEFAULT_TABLE table=test-table [id=d474d21140a2478d903398278296711b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:49.341514 27655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6022067a364828a756b4cee60ea6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:49.342468 27521 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6022067a364828a756b4cee60ea6bb (DEFAULT_TABLE table=test-table [id=d474d21140a2478d903398278296711b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:49.342765 27521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6022067a364828a756b4cee60ea6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:49.343302 27869 tablet_bootstrap.cc:492] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef: Bootstrap starting.
I20260928 12:34:49.344075 27869 tablet_bootstrap.cc:654] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:49.344537 27870 tablet_bootstrap.cc:492] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: Bootstrap starting.
I20260928 12:34:49.344843 27869 log.cc:824] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:49.345201 27870 tablet_bootstrap.cc:654] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:49.345708 27870 log.cc:824] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:49.345932 27869 tablet_bootstrap.cc:492] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef: No bootstrap required, opened a new log
I20260928 12:34:49.346038 27869 ts_tablet_manager.cc:1397] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:34:49.346169 27872 tablet_bootstrap.cc:492] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004: Bootstrap starting.
I20260928 12:34:49.346802 27872 tablet_bootstrap.cc:654] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:49.347368 27872 log.cc:824] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:49.348093 27869 raft_consensus.cc:348] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33931 } } peers { permanent_uuid: "942c5ce4907342c1b840288b774baec2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45881 } } peers { permanent_uuid: "772a963153eb411eb66479e25b769004" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43769 } }
I20260928 12:34:49.348209 27870 tablet_bootstrap.cc:492] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: No bootstrap required, opened a new log
I20260928 12:34:49.348209 27872 tablet_bootstrap.cc:492] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004: No bootstrap required, opened a new log
I20260928 12:34:49.348242 27869 raft_consensus.cc:374] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:49.348273 27870 ts_tablet_manager.cc:1397] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 12:34:49.348273 27872 ts_tablet_manager.cc:1397] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:34:49.348273 27869 raft_consensus.cc:729] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99e9bb00c5e4c9ca587bc23bd7660ef, State: Initialized, Role: FOLLOWER
I20260928 12:34:49.348404 27869 consensus_queue.cc:260] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33931 } } peers { permanent_uuid: "942c5ce4907342c1b840288b774baec2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45881 } } peers { permanent_uuid: "772a963153eb411eb66479e25b769004" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43769 } }
I20260928 12:34:49.348697 27855 heartbeater.cc:503] Master 127.8.13.62:33633 was elected leader, sending a full tablet report...
I20260928 12:34:49.348701 27869 ts_tablet_manager.cc:1428] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:34:49.350361 27872 raft_consensus.cc:348] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33931 } } peers { permanent_uuid: "942c5ce4907342c1b840288b774baec2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45881 } } peers { permanent_uuid: "772a963153eb411eb66479e25b769004" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43769 } }
I20260928 12:34:49.350361 27870 raft_consensus.cc:348] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33931 } } peers { permanent_uuid: "942c5ce4907342c1b840288b774baec2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45881 } } peers { permanent_uuid: "772a963153eb411eb66479e25b769004" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43769 } }
I20260928 12:34:49.350560 27870 raft_consensus.cc:374] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:49.350560 27872 raft_consensus.cc:374] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:49.350595 27872 raft_consensus.cc:729] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 772a963153eb411eb66479e25b769004, State: Initialized, Role: FOLLOWER
I20260928 12:34:49.350595 27870 raft_consensus.cc:729] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 942c5ce4907342c1b840288b774baec2, State: Initialized, Role: FOLLOWER
I20260928 12:34:49.350718 27872 consensus_queue.cc:260] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33931 } } peers { permanent_uuid: "942c5ce4907342c1b840288b774baec2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45881 } } peers { permanent_uuid: "772a963153eb411eb66479e25b769004" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43769 } }
I20260928 12:34:49.350718 27870 consensus_queue.cc:260] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33931 } } peers { permanent_uuid: "942c5ce4907342c1b840288b774baec2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45881 } } peers { permanent_uuid: "772a963153eb411eb66479e25b769004" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43769 } }
I20260928 12:34:49.351020 27870 ts_tablet_manager.cc:1428] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:34:49.351020 27872 ts_tablet_manager.cc:1428] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:34:49.351074 27587 heartbeater.cc:503] Master 127.8.13.62:33633 was elected leader, sending a full tablet report...
I20260928 12:34:49.351152 27721 heartbeater.cc:503] Master 127.8.13.62:33633 was elected leader, sending a full tablet report...
W20260928 12:34:49.400367 27722 tablet.cc:2369] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:34:49.448299 27877 raft_consensus.cc:482] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:34:49.448436 27877 raft_consensus.cc:504] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33931 } } peers { permanent_uuid: "942c5ce4907342c1b840288b774baec2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45881 } } peers { permanent_uuid: "772a963153eb411eb66479e25b769004" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43769 } }
I20260928 12:34:49.448757 27877 leader_election.cc:288] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e99e9bb00c5e4c9ca587bc23bd7660ef (127.8.13.3:33931), 942c5ce4907342c1b840288b774baec2 (127.8.13.2:45881)
I20260928 12:34:49.451675 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.449026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:39755 (local address 127.8.13.3:33931)
0928 12:34:49.449090 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.449092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.449100 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:34:49.449132 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.449133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.449145 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.449228 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:49.449393 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.449963 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.450962 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.451333 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.451391 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.451403 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.451564 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.451604 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:34:49.451675 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.449039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:45727 (local address 127.8.13.2:45881)
0928 12:34:49.449085 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.449087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.449219 (+ 132us) server_negotiation.cc:407] Connection header received
0928 12:34:49.449345 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.449346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.449357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.449423 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:49.449561 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.450064 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.450977 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.451347 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.451411 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.451421 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.451558 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.451600 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:34:49.451685 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.448943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:33931 (local address 127.8.13.1:39755)
0928 12:34:49.448986 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:49.448998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:49.449074 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:49.449240 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:49.449241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:49.449243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:49.449319 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.449322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.450050 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.450051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.450847 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.450849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.451410 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.451411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.451494 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.451497 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:49.451545 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:34:49.451589 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":2}
I20260928 12:34:49.451694 27878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.448943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:45881 (local address 127.8.13.1:45727)
0928 12:34:49.449162 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:49.449172 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:49.449242 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:49.449424 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:49.449425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:49.449427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:49.449485 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.449486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.450145 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.450146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.450881 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.450883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.451430 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.451431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.451489 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.451493 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:34:49.451538 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:34:49.451583 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":47,"thread_start_us":106,"threads_started":1}
I20260928 12:34:49.452067 27675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6022067a364828a756b4cee60ea6bb" candidate_uuid: "772a963153eb411eb66479e25b769004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "942c5ce4907342c1b840288b774baec2" is_pre_election: true
I20260928 12:34:49.452068 27809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6022067a364828a756b4cee60ea6bb" candidate_uuid: "772a963153eb411eb66479e25b769004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" is_pre_election: true
I20260928 12:34:49.452239 27675 raft_consensus.cc:2454] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 772a963153eb411eb66479e25b769004 in term 0.
I20260928 12:34:49.452236 27809 raft_consensus.cc:2454] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 772a963153eb411eb66479e25b769004 in term 0.
I20260928 12:34:49.452479 27477 leader_election.cc:302] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 772a963153eb411eb66479e25b769004, e99e9bb00c5e4c9ca587bc23bd7660ef; no voters:
I20260928 12:34:49.452656 27877 raft_consensus.cc:2781] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:34:49.452715 27877 raft_consensus.cc:482] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:34:49.452729 27877 raft_consensus.cc:3037] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:49.453454 27877 raft_consensus.cc:504] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33931 } } peers { permanent_uuid: "942c5ce4907342c1b840288b774baec2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45881 } } peers { permanent_uuid: "772a963153eb411eb66479e25b769004" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43769 } }
I20260928 12:34:49.453591 27877 leader_election.cc:288] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [CANDIDATE]: Term 1 election: Requested vote from peers e99e9bb00c5e4c9ca587bc23bd7660ef (127.8.13.3:33931), 942c5ce4907342c1b840288b774baec2 (127.8.13.2:45881)
I20260928 12:34:49.453748 27809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6022067a364828a756b4cee60ea6bb" candidate_uuid: "772a963153eb411eb66479e25b769004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef"
I20260928 12:34:49.453847 27675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6022067a364828a756b4cee60ea6bb" candidate_uuid: "772a963153eb411eb66479e25b769004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "942c5ce4907342c1b840288b774baec2"
I20260928 12:34:49.453864 27809 raft_consensus.cc:3037] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:49.453929 27675 raft_consensus.cc:3037] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:49.454809 27809 raft_consensus.cc:2454] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 772a963153eb411eb66479e25b769004 in term 1.
I20260928 12:34:49.454809 27675 raft_consensus.cc:2454] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 772a963153eb411eb66479e25b769004 in term 1.
I20260928 12:34:49.454994 27476 leader_election.cc:302] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 772a963153eb411eb66479e25b769004, 942c5ce4907342c1b840288b774baec2, e99e9bb00c5e4c9ca587bc23bd7660ef; no voters:
I20260928 12:34:49.455087 27877 raft_consensus.cc:2781] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:49.455264 27877 raft_consensus.cc:686] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [term 1 LEADER]: Becoming Leader. State: Replica: 772a963153eb411eb66479e25b769004, State: Running, Role: LEADER
I20260928 12:34:49.455365 27877 consensus_queue.cc:237] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33931 } } peers { permanent_uuid: "942c5ce4907342c1b840288b774baec2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45881 } } peers { permanent_uuid: "772a963153eb411eb66479e25b769004" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43769 } }
I20260928 12:34:49.456249 27396 catalog_manager.cc:5495] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 reported cstate change: term changed from 0 to 1, leader changed from <none> to 772a963153eb411eb66479e25b769004 (127.8.13.1). New cstate: current_term: 1 leader_uuid: "772a963153eb411eb66479e25b769004" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "942c5ce4907342c1b840288b774baec2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "772a963153eb411eb66479e25b769004" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 43769 } health_report { overall_health: HEALTHY } } }
I20260928 12:34:49.482453 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.480072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.8.13.1:43769)
0928 12:34:49.480117 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.480118 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.480144 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:34:49.480235 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.480237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.480258 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.480319 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:34:49.480457 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.480926 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.481671 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.481895 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.481953 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.482125 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:34:49.482232 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:34:49.482304 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.482341 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.482364 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:34:49.484215 27809 raft_consensus.cc:1261] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef [term 1 FOLLOWER]: Refusing update from remote peer 772a963153eb411eb66479e25b769004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:49.484205 27675 raft_consensus.cc:1261] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2 [term 1 FOLLOWER]: Refusing update from remote peer 772a963153eb411eb66479e25b769004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:49.484558 27877 consensus_queue.cc:1035] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99e9bb00c5e4c9ca587bc23bd7660ef" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:49.484699 27879 consensus_queue.cc:1035] T 5c6022067a364828a756b4cee60ea6bb P 772a963153eb411eb66479e25b769004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "942c5ce4907342c1b840288b774baec2" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:49.487685 27883 mvcc.cc:205] Tried to move back new op lower bound from 7334293051324567552 to 7334293051209838592. Current Snapshot: MvccSnapshot[applied={T|T < 7334293051324567552}]
I20260928 12:34:49.489456 27396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:49.489542 27396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:49.490236 27396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dcb4d01c34e4c62ad02c989ef6db6ef: Sending DeleteTablet for 3 replicas of tablet 5c6022067a364828a756b4cee60ea6bb
I20260928 12:34:49.490576 27521 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c6022067a364828a756b4cee60ea6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:49 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:42168
I20260928 12:34:49.490572 27789 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c6022067a364828a756b4cee60ea6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:49 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:50870
I20260928 12:34:49.490780 27892 tablet_replica.cc:330] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef: stopping tablet replica
I20260928 12:34:49.490818 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 27458
I20260928 12:34:49.490870 27892 raft_consensus.cc:2229] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:34:49.490967 27892 raft_consensus.cc:2258] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:49.491662 27892 ts_tablet_manager.cc:1905] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:49.493409 27892 ts_tablet_manager.cc:1918] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:49.493479 27892 log.cc:1196] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/5c6022067a364828a756b4cee60ea6bb
I20260928 12:34:49.493736 27892 ts_tablet_manager.cc:1939] T 5c6022067a364828a756b4cee60ea6bb P e99e9bb00c5e4c9ca587bc23bd7660ef: Deleting consensus metadata
I20260928 12:34:49.494256 27383 catalog_manager.cc:4841] TS e99e9bb00c5e4c9ca587bc23bd7660ef (127.8.13.3:33931): tablet 5c6022067a364828a756b4cee60ea6bb (table test-table [id=d474d21140a2478d903398278296711b]) successfully deleted
I20260928 12:34:49.494650 27655 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c6022067a364828a756b4cee60ea6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:49 UTC) from {username='kudu', principal='kudu/127.8.13.62@KRBTEST.COM'} at 127.0.0.1:33364
I20260928 12:34:49.494925 27894 tablet_replica.cc:330] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: stopping tablet replica
I20260928 12:34:49.495030 27894 raft_consensus.cc:2229] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:34:49.495152 27894 raft_consensus.cc:2258] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:49.495536 27894 ts_tablet_manager.cc:1905] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:49.496968 27894 ts_tablet_manager.cc:1918] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:49.497030 27894 log.cc:1196] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/5c6022067a364828a756b4cee60ea6bb
I20260928 12:34:49.497233 27894 ts_tablet_manager.cc:1939] T 5c6022067a364828a756b4cee60ea6bb P 942c5ce4907342c1b840288b774baec2: Deleting consensus metadata
W20260928 12:34:49.497740 27380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:43769 (error 108)
I20260928 12:34:49.497921 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 27592
I20260928 12:34:49.498405 27382 catalog_manager.cc:4841] TS 942c5ce4907342c1b840288b774baec2 (127.8.13.2:45881): tablet 5c6022067a364828a756b4cee60ea6bb (table test-table [id=d474d21140a2478d903398278296711b]) successfully deleted
I20260928 12:34:49.498697 27864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.498581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:43769 (local address 127.0.0.1:42190)
0928 12:34:49.498622 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:49.498662 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 12:34:49.498809 27380 catalog_manager.cc:4584] TS 772a963153eb411eb66479e25b769004 (127.8.13.1:43769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c6022067a364828a756b4cee60ea6bb: Network error: Client connection negotiation failed: client connection to 127.8.13.1:43769: connect: Connection refused (error 111)
I20260928 12:34:49.502672 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 27726
I20260928 12:34:49.507197 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 27364
2026-09-28T12:34:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1866 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T12:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:49Z Disabled control of system clock
I20260928 12:34:49.532048 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:41311
--webserver_interface=127.8.13.62
--webserver_port=0
--builtin_ntp_servers=127.8.13.20:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:41311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:34:49.612830 27899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:49.613059 27899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:49.613078 27899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:49.614192 27899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:49.614228 27899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:49.614238 27899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:49.614249 27899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:49.614259 27899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:49.615551 27899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:41311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:41311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:49.615747 27899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:49.615979 27899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:49.620468 27907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:49.620474 27904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:49.620476 27905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:49.620515 27899 server_base.cc:1034] running on GCE node
I20260928 12:34:49.620867 27899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:49.621133 27899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:49.622259 27899 hybrid_clock.cc:648] HybridClock initialized: now 1790598889622243 us; error 32 us; skew 500 ppm
I20260928 12:34:49.623489 27899 webserver.cc:460] Webserver started at http://127.8.13.62:40139/ using document root <none> and password file <none>
I20260928 12:34:49.623713 27899 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:49.623756 27899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:49.623878 27899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:49.624917 27899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "64f341aa4b5c4ef6bfca265b052db06d"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "41ae2df71ea596133c66a64717f05d41"
server_key_iv: "fce1661ff8e796c239e36117eccb8114"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.625240 27899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "64f341aa4b5c4ef6bfca265b052db06d"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "41ae2df71ea596133c66a64717f05d41"
server_key_iv: "fce1661ff8e796c239e36117eccb8114"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.626467 27899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:34:49.627125 27913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.627261 27899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:34:49.627334 27899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "64f341aa4b5c4ef6bfca265b052db06d"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "41ae2df71ea596133c66a64717f05d41"
server_key_iv: "fce1661ff8e796c239e36117eccb8114"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.627400 27899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:49.653437 27899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:49.653955 27899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:49.654105 27899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:49.657857 27899 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:41311
I20260928 12:34:49.657894 27965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:41311 every 8 connection(s)
I20260928 12:34:49.658239 27899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:49.659215 27966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:49.661958 27966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d: Bootstrap starting.
I20260928 12:34:49.662546 27966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:49.662822 27966 log.cc:824] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:49.663476 27966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d: No bootstrap required, opened a new log
I20260928 12:34:49.665102 27966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f341aa4b5c4ef6bfca265b052db06d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41311 } }
I20260928 12:34:49.665246 27966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:49.665277 27966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f341aa4b5c4ef6bfca265b052db06d, State: Initialized, Role: FOLLOWER
I20260928 12:34:49.665423 27966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f341aa4b5c4ef6bfca265b052db06d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41311 } }
I20260928 12:34:49.665485 27966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:49.665508 27966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:49.665551 27966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:49.666121 27966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f341aa4b5c4ef6bfca265b052db06d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41311 } }
I20260928 12:34:49.666234 27966 leader_election.cc:302] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64f341aa4b5c4ef6bfca265b052db06d; no voters:
I20260928 12:34:49.666318 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 27899
I20260928 12:34:49.666419 27966 leader_election.cc:288] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:49.666426 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:49.666499 27969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:49.666653 8244 external_mini_cluster.cc:1428] Setting key 6b8407dd348fbc39164c8c6d3dda776b
I20260928 12:34:49.666674 27969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [term 1 LEADER]: Becoming Leader. State: Replica: 64f341aa4b5c4ef6bfca265b052db06d, State: Running, Role: LEADER
I20260928 12:34:49.666674 27966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:49.666819 27969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f341aa4b5c4ef6bfca265b052db06d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41311 } }
I20260928 12:34:49.667160 27969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64f341aa4b5c4ef6bfca265b052db06d. Latest consensus state: current_term: 1 leader_uuid: "64f341aa4b5c4ef6bfca265b052db06d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f341aa4b5c4ef6bfca265b052db06d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41311 } } }
I20260928 12:34:49.667237 27969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:49.667362 27971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64f341aa4b5c4ef6bfca265b052db06d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f341aa4b5c4ef6bfca265b052db06d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41311 } } }
I20260928 12:34:49.667436 27971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d [sys.catalog]: This master's current role is: LEADER
W20260928 12:34:49.668521 27986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:34:49.668584 27986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:34:49.668668 27978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:49.668860 27978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:49.672601 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.667273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.8.13.62:41311)
0928 12:34:49.667498 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.667507 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.667736 (+ 229us) server_negotiation.cc:407] Connection header received
0928 12:34:49.667878 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.667886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.667976 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.668153 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:49.668521 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.669792 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.670636 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.671080 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.671197 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.671937 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:49.671941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:49.671942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:49.672123 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:49.672159 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:49.672171 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:49.672227 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.672310 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.672394 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260928 12:34:49.678181 27978 catalog_manager.cc:1347] Generated new cluster ID: 9f15e5d8e1d448b49054d23203c4dc87
I20260928 12:34:49.678268 27978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:49.695363 27978 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:49.696050 27978 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:49.699075 27978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d: Generated new TSK 0
I20260928 12:34:49.699249 27978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:34:49.704908 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.1:0
--local_ip_for_outbound_sockets=127.8.13.1
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41311
--builtin_ntp_servers=127.8.13.20:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:34:49.788419 27990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:49.788640 27990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:49.788666 27990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:49.790026 27990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:49.790073 27990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:49.790102 27990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.1
I20260928 12:34:49.791529 27990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.1
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.1
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:49.791781 27990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:49.792147 27990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:49.797355 27995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:49.797359 27996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:49.797415 27990 server_base.cc:1034] running on GCE node
W20260928 12:34:49.797381 27998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:49.797884 27990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:49.798149 27990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:49.799324 27990 hybrid_clock.cc:648] HybridClock initialized: now 1790598889799308 us; error 31 us; skew 500 ppm
I20260928 12:34:49.800752 27990 webserver.cc:460] Webserver started at http://127.8.13.1:39589/ using document root <none> and password file <none>
I20260928 12:34:49.800999 27990 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:49.801046 27990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:49.801170 27990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:49.802325 27990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/data/instance:
uuid: "a755f46f2cbf4fbe800129450b16b580"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "53089812273998497765a492bf9a8982"
server_key_iv: "175a5cff32bc2767bbff1108a4f115db"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.802650 27990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance:
uuid: "a755f46f2cbf4fbe800129450b16b580"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "53089812273998497765a492bf9a8982"
server_key_iv: "175a5cff32bc2767bbff1108a4f115db"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.803957 27990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:34:49.804795 28004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.804989 27990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:49.805074 27990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/wal
uuid: "a755f46f2cbf4fbe800129450b16b580"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "53089812273998497765a492bf9a8982"
server_key_iv: "175a5cff32bc2767bbff1108a4f115db"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.805140 27990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:49.823467 27990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:49.823772 27990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:49.823902 27990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:49.824105 27990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:49.824431 27990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:49.824465 27990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.824496 27990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:49.824513 27990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.829910 27990 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.1:40523
I20260928 12:34:49.829954 28117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.1:40523 every 8 connection(s)
I20260928 12:34:49.830271 27990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/data/info.pb
I20260928 12:34:49.835533 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.830972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.1:39453 (local address 127.8.13.62:41311)
0928 12:34:49.831033 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.831035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.831330 (+ 295us) server_negotiation.cc:407] Connection header received
0928 12:34:49.831815 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.831816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.831829 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.831881 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:49.832553 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.833076 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.834184 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.834399 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.834453 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.835277 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:49.835278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:49.835278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:49.835284 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:49.835297 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:49.835301 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:49.835341 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.835423 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.835461 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:34:49.835734 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.831003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:41311 (local address 127.8.13.1:39453)
0928 12:34:49.831215 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:49.831251 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:49.831728 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:49.831979 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:49.831982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:49.831990 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:49.832486 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.832490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.833185 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.833186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.834103 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.834106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.834470 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.834471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.834586 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.835145 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:34:49.835145 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:49.835178 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:34:49.835206 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:49.835361 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:49.835364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:49.835367 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:49.835420 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:34:49.835515 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260928 12:34:49.836365 28118 heartbeater.cc:347] Connected to a master server at 127.8.13.62:41311
I20260928 12:34:49.836477 28118 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:49.836639 28118 heartbeater.cc:511] Master 127.8.13.62:41311 requested a full tablet report, sending...
I20260928 12:34:49.837139 27930 ts_manager.cc:194] Registered new tserver with Master: a755f46f2cbf4fbe800129450b16b580 (127.8.13.1:40523)
I20260928 12:34:49.838050 27930 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.1:39453
I20260928 12:34:49.840022 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 27990
I20260928 12:34:49.840108 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-0/wal/instance
I20260928 12:34:49.840267 8244 external_mini_cluster.cc:1428] Setting key 7922b2380d13b2635d4f8eb895b0a3a8
I20260928 12:34:49.841326 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.2:0
--local_ip_for_outbound_sockets=127.8.13.2
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41311
--builtin_ntp_servers=127.8.13.20:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:34:49.922580 28121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:49.922786 28121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:49.922816 28121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:49.924283 28121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:49.924335 28121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:49.924371 28121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.2
I20260928 12:34:49.926213 28121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.2
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.2
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:49.926452 28121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:49.926728 28121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:49.931449 28127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:49.931437 28129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:49.931437 28126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:49.931649 28121 server_base.cc:1034] running on GCE node
I20260928 12:34:49.931845 28121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:49.932053 28121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:49.933190 28121 hybrid_clock.cc:648] HybridClock initialized: now 1790598889933176 us; error 30 us; skew 500 ppm
I20260928 12:34:49.934409 28121 webserver.cc:460] Webserver started at http://127.8.13.2:42597/ using document root <none> and password file <none>
I20260928 12:34:49.934639 28121 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:49.934686 28121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:49.934810 28121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:49.935878 28121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/data/instance:
uuid: "a24b15dd270b42fbb434cfd8459d8074"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "fa66fca2e932fc471770feafc50fe648"
server_key_iv: "318cbeda6a9472368d0a8c0d5dc06763"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.936203 28121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance:
uuid: "a24b15dd270b42fbb434cfd8459d8074"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "fa66fca2e932fc471770feafc50fe648"
server_key_iv: "318cbeda6a9472368d0a8c0d5dc06763"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.937393 28121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:34:49.938078 28135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.938213 28121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:49.938287 28121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/wal
uuid: "a24b15dd270b42fbb434cfd8459d8074"
format_stamp: "Formatted at 2026-09-28 12:34:49 on dist-test-slave-g31s"
server_key: "fa66fca2e932fc471770feafc50fe648"
server_key_iv: "318cbeda6a9472368d0a8c0d5dc06763"
server_key_version: "encryptionkey@0"
I20260928 12:34:49.938351 28121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:49.950865 28121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:49.951172 28121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:49.951297 28121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:49.951531 28121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:49.951871 28121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:49.951906 28121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.951930 28121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:49.951953 28121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:49.957547 28121 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.2:34093
I20260928 12:34:49.957590 28248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.2:34093 every 8 connection(s)
I20260928 12:34:49.957950 28121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/data/info.pb
I20260928 12:34:49.963229 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.958655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.2:41579 (local address 127.8.13.62:41311)
0928 12:34:49.958717 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:49.958719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:49.958954 (+ 235us) server_negotiation.cc:407] Connection header received
0928 12:34:49.959515 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:49.959516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:49.959528 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:49.959581 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:49.960284 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.960783 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.961763 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.962029 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.962099 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.962952 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:49.962953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:49.962953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:49.962959 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:49.962974 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:49.962979 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:49.963022 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:49.963106 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 12:34:49.963142 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:34:49.963470 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:49.958647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:41311 (local address 127.8.13.2:41579)
0928 12:34:49.958829 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:49.958866 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:49.959374 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:49.959654 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:49.959658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:49.959666 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:49.960184 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.960189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.960873 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.960874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.961671 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:49.961673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:49.962118 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:49.962119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:49.962249 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:49.962752 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:34:49.962753 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:49.962784 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:34:49.962810 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:49.963034 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:49.963041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:49.963045 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:34:49.963106 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 12:34:49.963222 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260928 12:34:49.963932 28249 heartbeater.cc:347] Connected to a master server at 127.8.13.62:41311
I20260928 12:34:49.964032 28249 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:49.964212 28249 heartbeater.cc:511] Master 127.8.13.62:41311 requested a full tablet report, sending...
I20260928 12:34:49.964556 27930 ts_manager.cc:194] Registered new tserver with Master: a24b15dd270b42fbb434cfd8459d8074 (127.8.13.2:34093)
I20260928 12:34:49.964926 27930 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.2:41579
I20260928 12:34:49.966110 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 28121
I20260928 12:34:49.966212 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/wal/instance
I20260928 12:34:49.966413 8244 external_mini_cluster.cc:1428] Setting key d04cd688c318d66d3d5ad485ef25cc62
I20260928 12:34:49.967443 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.3:0
--local_ip_for_outbound_sockets=127.8.13.3
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41311
--builtin_ntp_servers=127.8.13.20:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:34:50.047640 28252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:50.047863 28252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:50.047884 28252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:50.049134 28252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:50.049189 28252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:50.049217 28252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.3
I20260928 12:34:50.050717 28252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.3
--webserver_port=0
--tserver_master_addrs=127.8.13.62:41311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.3
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:50.050968 28252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:50.051309 28252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:50.056195 28257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:50.056187 28258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:50.056255 28252 server_base.cc:1034] running on GCE node
W20260928 12:34:50.056187 28260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:50.056591 28252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:50.056809 28252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:50.057938 28252 hybrid_clock.cc:648] HybridClock initialized: now 1790598890057927 us; error 28 us; skew 500 ppm
I20260928 12:34:50.059162 28252 webserver.cc:460] Webserver started at http://127.8.13.3:34853/ using document root <none> and password file <none>
I20260928 12:34:50.059397 28252 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:50.059444 28252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:50.059561 28252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:50.060613 28252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/data/instance:
uuid: "54a521a23d8342258547396d5c05c636"
format_stamp: "Formatted at 2026-09-28 12:34:50 on dist-test-slave-g31s"
server_key: "b6261ee1e5932824b2244bb05c504de1"
server_key_iv: "358d33d30450ee21dce0223ddd423ec6"
server_key_version: "encryptionkey@0"
I20260928 12:34:50.060928 28252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance:
uuid: "54a521a23d8342258547396d5c05c636"
format_stamp: "Formatted at 2026-09-28 12:34:50 on dist-test-slave-g31s"
server_key: "b6261ee1e5932824b2244bb05c504de1"
server_key_iv: "358d33d30450ee21dce0223ddd423ec6"
server_key_version: "encryptionkey@0"
I20260928 12:34:50.062201 28252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:34:50.062883 28266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:50.063045 28252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:34:50.063123 28252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/wal
uuid: "54a521a23d8342258547396d5c05c636"
format_stamp: "Formatted at 2026-09-28 12:34:50 on dist-test-slave-g31s"
server_key: "b6261ee1e5932824b2244bb05c504de1"
server_key_iv: "358d33d30450ee21dce0223ddd423ec6"
server_key_version: "encryptionkey@0"
I20260928 12:34:50.063190 28252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:50.111550 28252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:50.111852 28252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:50.111963 28252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:50.112198 28252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:34:50.112501 28252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:34:50.112536 28252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:50.112558 28252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:34:50.112581 28252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:50.117859 28252 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.3:42215
I20260928 12:34:50.117926 28379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.3:42215 every 8 connection(s)
I20260928 12:34:50.118220 28252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/data/info.pb
I20260928 12:34:50.122802 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.118883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:52129 (local address 127.8.13.62:41311)
0928 12:34:50.118988 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:50.118990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:50.119144 (+ 154us) server_negotiation.cc:407] Connection header received
0928 12:34:50.119613 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:50.119614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:50.119624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:50.119674 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:50.120273 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.120765 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.121653 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.121844 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.121885 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.122581 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:50.122583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:50.122583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:50.122587 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:50.122600 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:50.122603 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:50.122640 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:50.122699 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 12:34:50.122726 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:34:50.122954 28382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.118853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:41311 (local address 127.8.13.3:52129)
0928 12:34:50.119035 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:50.119069 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:50.119539 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:50.119726 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:50.119729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:50.119737 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:34:50.120213 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.120217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.120830 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.120831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.121594 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.121596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.121908 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.121908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.122013 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.122450 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:34:50.122451 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:34:50.122479 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:34:50.122502 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:34:50.122647 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:34:50.122649 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:34:50.122652 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:50.122696 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:34:50.122772 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260928 12:34:50.122948 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 28252
I20260928 12:34:50.123046 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/wal/instance
I20260928 12:34:50.123251 8244 external_mini_cluster.cc:1428] Setting key 9c0c34cbcfb9020e980e619a767a67cb
I20260928 12:34:50.123358 28380 heartbeater.cc:347] Connected to a master server at 127.8.13.62:41311
I20260928 12:34:50.123450 28380 heartbeater.cc:464] Registering TS with master...
I20260928 12:34:50.123638 28380 heartbeater.cc:511] Master 127.8.13.62:41311 requested a full tablet report, sending...
I20260928 12:34:50.124006 27929 ts_manager.cc:194] Registered new tserver with Master: 54a521a23d8342258547396d5c05c636 (127.8.13.3:42215)
I20260928 12:34:50.124363 27929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.3:52129
I20260928 12:34:50.134928 8244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:34:50.139106 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.136242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.8.13.62:41311)
0928 12:34:50.136293 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:50.136296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:50.136370 (+ 74us) server_negotiation.cc:407] Connection header received
0928 12:34:50.136415 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:50.136415 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:50.136427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:50.136479 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:50.136625 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.137119 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.137938 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.138155 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.138223 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.138900 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:50.138901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:50.138902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:50.138907 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:50.138921 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:50.138925 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:50.138968 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:50.139009 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 12:34:50.139038 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:34:50.140741 27929 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:34:50.142410 27929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:34:50.148643 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.144475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:34093 (local address 127.0.0.1:50864)
0928 12:34:50.144672 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:50.144714 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:50.144782 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:50.145078 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:50.145080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:50.145083 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:50.145160 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.145162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.146539 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.146540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.147612 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.147615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.148313 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.148314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.148406 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.148422 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 12:34:50.148493 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:34:50.148550 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20260928 12:34:50.148658 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.144724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.8.13.2:34093)
0928 12:34:50.144902 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:50.144908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:50.144919 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:34:50.144961 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:50.144963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:50.144979 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:50.145070 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:50.145244 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.146455 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.147691 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.148227 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.148299 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.148399 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:50.148525 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 12:34:50.148571 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260928 12:34:50.149538 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.144475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:40523 (local address 127.0.0.1:46386)
0928 12:34:50.144718 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:50.144725 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:50.144782 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:50.145180 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:50.145180 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:50.145182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:50.145213 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.145214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.147713 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.147715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.148376 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.148377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.149287 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.149288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.149352 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.149357 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:34:50.149414 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:34:50.149459 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260928 12:34:50.149545 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.144810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.8.13.1:40523)
0928 12:34:50.144984 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:50.144988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:50.144997 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:34:50.145033 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:50.145035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:50.145048 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:50.145118 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:50.145284 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.146455 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.148463 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.149217 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.149286 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.149376 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:50.149427 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 12:34:50.149465 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260928 12:34:50.149676 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.144478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.3:42215 (local address 127.0.0.1:57924)
0928 12:34:50.144873 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:50.144880 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:50.144916 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:50.145203 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:50.145205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:50.145207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:50.145263 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.145265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.146283 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.146285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.147612 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.147615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.149337 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.149339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.149498 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.149504 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:34:50.149558 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:34:50.149605 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":11,"thread_start_us":87,"threads_started":1}
I20260928 12:34:50.149901 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.144637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.8.13.3:42215)
0928 12:34:50.144992 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:50.144995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:50.145006 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:34:50.145039 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:50.145041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:50.145054 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:50.145119 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:50.145335 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.146214 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.148454 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.149151 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.149209 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.149301 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:50.149746 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 12:34:50.149791 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":86,"threads_started":1}
I20260928 12:34:50.149914 28183 tablet_service.cc:1467] Processing CreateTablet for tablet a9004e31fd5f47aba6056967135cb4f9 (DEFAULT_TABLE table=test-table [id=ef6606f542de4b6caaf85329bb742655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:50.150301 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9004e31fd5f47aba6056967135cb4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:50.150498 28052 tablet_service.cc:1467] Processing CreateTablet for tablet a9004e31fd5f47aba6056967135cb4f9 (DEFAULT_TABLE table=test-table [id=ef6606f542de4b6caaf85329bb742655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:50.150635 28314 tablet_service.cc:1467] Processing CreateTablet for tablet a9004e31fd5f47aba6056967135cb4f9 (DEFAULT_TABLE table=test-table [id=ef6606f542de4b6caaf85329bb742655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:34:50.150835 28052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9004e31fd5f47aba6056967135cb4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:50.150985 28314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9004e31fd5f47aba6056967135cb4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:50.153455 28394 tablet_bootstrap.cc:492] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636: Bootstrap starting.
I20260928 12:34:50.153889 28395 tablet_bootstrap.cc:492] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074: Bootstrap starting.
I20260928 12:34:50.154191 28396 tablet_bootstrap.cc:492] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580: Bootstrap starting.
I20260928 12:34:50.154326 28394 tablet_bootstrap.cc:654] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:50.154497 28395 tablet_bootstrap.cc:654] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:50.154703 28394 log.cc:824] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:50.154781 28396 tablet_bootstrap.cc:654] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:50.154798 28395 log.cc:824] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:50.155222 28396 log.cc:824] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:50.155484 28395 tablet_bootstrap.cc:492] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074: No bootstrap required, opened a new log
I20260928 12:34:50.155479 28394 tablet_bootstrap.cc:492] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636: No bootstrap required, opened a new log
I20260928 12:34:50.155552 28395 ts_tablet_manager.cc:1397] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:34:50.155552 28394 ts_tablet_manager.cc:1397] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:34:50.155869 28396 tablet_bootstrap.cc:492] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580: No bootstrap required, opened a new log
I20260928 12:34:50.155927 28396 ts_tablet_manager.cc:1397] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:34:50.157944 28396 raft_consensus.cc:348] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a755f46f2cbf4fbe800129450b16b580" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40523 } } peers { permanent_uuid: "54a521a23d8342258547396d5c05c636" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42215 } } peers { permanent_uuid: "a24b15dd270b42fbb434cfd8459d8074" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34093 } }
I20260928 12:34:50.158108 28396 raft_consensus.cc:374] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:50.158140 28396 raft_consensus.cc:729] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a755f46f2cbf4fbe800129450b16b580, State: Initialized, Role: FOLLOWER
I20260928 12:34:50.158102 28394 raft_consensus.cc:348] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a755f46f2cbf4fbe800129450b16b580" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40523 } } peers { permanent_uuid: "54a521a23d8342258547396d5c05c636" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42215 } } peers { permanent_uuid: "a24b15dd270b42fbb434cfd8459d8074" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34093 } }
I20260928 12:34:50.158104 28395 raft_consensus.cc:348] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a755f46f2cbf4fbe800129450b16b580" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40523 } } peers { permanent_uuid: "54a521a23d8342258547396d5c05c636" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42215 } } peers { permanent_uuid: "a24b15dd270b42fbb434cfd8459d8074" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34093 } }
I20260928 12:34:50.158291 28394 raft_consensus.cc:374] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:50.158291 28395 raft_consensus.cc:374] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:50.158334 28395 raft_consensus.cc:729] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24b15dd270b42fbb434cfd8459d8074, State: Initialized, Role: FOLLOWER
I20260928 12:34:50.158334 28394 raft_consensus.cc:729] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a521a23d8342258547396d5c05c636, State: Initialized, Role: FOLLOWER
I20260928 12:34:50.158316 28396 consensus_queue.cc:260] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a755f46f2cbf4fbe800129450b16b580" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40523 } } peers { permanent_uuid: "54a521a23d8342258547396d5c05c636" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42215 } } peers { permanent_uuid: "a24b15dd270b42fbb434cfd8459d8074" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34093 } }
I20260928 12:34:50.158514 28395 consensus_queue.cc:260] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a755f46f2cbf4fbe800129450b16b580" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40523 } } peers { permanent_uuid: "54a521a23d8342258547396d5c05c636" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42215 } } peers { permanent_uuid: "a24b15dd270b42fbb434cfd8459d8074" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34093 } }
I20260928 12:34:50.158515 28394 consensus_queue.cc:260] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a755f46f2cbf4fbe800129450b16b580" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40523 } } peers { permanent_uuid: "54a521a23d8342258547396d5c05c636" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42215 } } peers { permanent_uuid: "a24b15dd270b42fbb434cfd8459d8074" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34093 } }
I20260928 12:34:50.158636 28396 ts_tablet_manager.cc:1428] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:34:50.158769 28118 heartbeater.cc:503] Master 127.8.13.62:41311 was elected leader, sending a full tablet report...
I20260928 12:34:50.158771 28395 ts_tablet_manager.cc:1428] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 12:34:50.158772 28394 ts_tablet_manager.cc:1428] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 12:34:50.158881 28380 heartbeater.cc:503] Master 127.8.13.62:41311 was elected leader, sending a full tablet report...
I20260928 12:34:50.158891 28249 heartbeater.cc:503] Master 127.8.13.62:41311 was elected leader, sending a full tablet report...
I20260928 12:34:50.175997 28401 raft_consensus.cc:482] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:34:50.176160 28401 raft_consensus.cc:504] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a755f46f2cbf4fbe800129450b16b580" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40523 } } peers { permanent_uuid: "54a521a23d8342258547396d5c05c636" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42215 } } peers { permanent_uuid: "a24b15dd270b42fbb434cfd8459d8074" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34093 } }
I20260928 12:34:50.176540 28401 leader_election.cc:288] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a755f46f2cbf4fbe800129450b16b580 (127.8.13.1:40523), a24b15dd270b42fbb434cfd8459d8074 (127.8.13.2:34093)
I20260928 12:34:50.179575 28382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.176715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.2:34093 (local address 127.8.13.3:38169)
0928 12:34:50.176850 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:50.176862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:50.176914 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:50.177052 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:50.177053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:50.177054 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:50.177118 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.177119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.177982 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.177982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.178748 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.178750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.179309 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.179310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.179384 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.179395 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 12:34:50.179445 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:34:50.179491 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260928 12:34:50.179913 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.176721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:40523 (local address 127.8.13.3:40561)
0928 12:34:50.177203 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:50.177211 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:34:50.177245 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:34:50.177365 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:34:50.177365 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:34:50.177366 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:34:50.177408 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.177409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.178167 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.178169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.179004 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:34:50.179007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.179670 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.179671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:34:50.179737 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.179740 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:34:50.179783 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:34:50.179821 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":156,"thread_start_us":144,"threads_started":1}
I20260928 12:34:50.179903 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.176817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:40561 (local address 127.8.13.1:40523)
0928 12:34:50.176866 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:50.176867 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:50.177263 (+ 396us) server_negotiation.cc:407] Connection header received
0928 12:34:50.177293 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:50.177293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:50.177305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:50.177362 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:50.177457 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.178096 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.179092 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.179563 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.179621 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.179631 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:50.179801 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 12:34:50.179842 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:34:50.180229 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.176793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.3:38169 (local address 127.8.13.2:34093)
0928 12:34:50.176849 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:50.176851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:50.176911 (+ 60us) server_negotiation.cc:407] Connection header received
0928 12:34:50.176971 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:50.176972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:50.176986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:50.177049 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:34:50.177169 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.177906 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.178853 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.179238 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.179294 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.179306 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:50.179473 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 12:34:50.179558 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:34:50.180222 28072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9004e31fd5f47aba6056967135cb4f9" candidate_uuid: "54a521a23d8342258547396d5c05c636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a755f46f2cbf4fbe800129450b16b580" is_pre_election: true
I20260928 12:34:50.180397 28072 raft_consensus.cc:2454] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54a521a23d8342258547396d5c05c636 in term 0.
I20260928 12:34:50.180506 28203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9004e31fd5f47aba6056967135cb4f9" candidate_uuid: "54a521a23d8342258547396d5c05c636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a24b15dd270b42fbb434cfd8459d8074" is_pre_election: true
I20260928 12:34:50.180655 28203 raft_consensus.cc:2454] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54a521a23d8342258547396d5c05c636 in term 0.
I20260928 12:34:50.180644 28268 leader_election.cc:302] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54a521a23d8342258547396d5c05c636, a755f46f2cbf4fbe800129450b16b580; no voters:
I20260928 12:34:50.181880 28401 raft_consensus.cc:2781] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:34:50.181967 28401 raft_consensus.cc:482] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:34:50.182037 28401 raft_consensus.cc:3037] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:50.182786 28401 raft_consensus.cc:504] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a755f46f2cbf4fbe800129450b16b580" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40523 } } peers { permanent_uuid: "54a521a23d8342258547396d5c05c636" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42215 } } peers { permanent_uuid: "a24b15dd270b42fbb434cfd8459d8074" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34093 } }
I20260928 12:34:50.182935 28401 leader_election.cc:288] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [CANDIDATE]: Term 1 election: Requested vote from peers a755f46f2cbf4fbe800129450b16b580 (127.8.13.1:40523), a24b15dd270b42fbb434cfd8459d8074 (127.8.13.2:34093)
I20260928 12:34:50.183125 28203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9004e31fd5f47aba6056967135cb4f9" candidate_uuid: "54a521a23d8342258547396d5c05c636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a24b15dd270b42fbb434cfd8459d8074"
I20260928 12:34:50.183215 28203 raft_consensus.cc:3037] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:50.183317 28072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9004e31fd5f47aba6056967135cb4f9" candidate_uuid: "54a521a23d8342258547396d5c05c636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a755f46f2cbf4fbe800129450b16b580"
I20260928 12:34:50.183430 28072 raft_consensus.cc:3037] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:50.184171 28203 raft_consensus.cc:2454] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54a521a23d8342258547396d5c05c636 in term 1.
I20260928 12:34:50.184376 28072 raft_consensus.cc:2454] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54a521a23d8342258547396d5c05c636 in term 1.
I20260928 12:34:50.184396 28267 leader_election.cc:302] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54a521a23d8342258547396d5c05c636, a24b15dd270b42fbb434cfd8459d8074; no voters:
I20260928 12:34:50.184546 28401 raft_consensus.cc:2781] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:50.184723 28401 raft_consensus.cc:686] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 1 LEADER]: Becoming Leader. State: Replica: 54a521a23d8342258547396d5c05c636, State: Running, Role: LEADER
I20260928 12:34:50.184836 28401 consensus_queue.cc:237] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a755f46f2cbf4fbe800129450b16b580" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40523 } } peers { permanent_uuid: "54a521a23d8342258547396d5c05c636" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42215 } } peers { permanent_uuid: "a24b15dd270b42fbb434cfd8459d8074" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34093 } }
I20260928 12:34:50.185695 27930 catalog_manager.cc:5495] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54a521a23d8342258547396d5c05c636 (127.8.13.3). New cstate: current_term: 1 leader_uuid: "54a521a23d8342258547396d5c05c636" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a755f46f2cbf4fbe800129450b16b580" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54a521a23d8342258547396d5c05c636" member_type: VOTER last_known_addr { host: "127.8.13.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a24b15dd270b42fbb434cfd8459d8074" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34093 } health_report { overall_health: UNKNOWN } } }
I20260928 12:34:50.198652 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.196334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.8.13.3:42215)
0928 12:34:50.196383 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:50.196385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:50.196393 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:34:50.196479 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:50.196480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:50.196499 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:50.196567 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:34:50.196679 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.197150 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.197924 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:50.198123 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:50.198177 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:50.198320 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:34:50.198423 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:34:50.198493 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:50.198533 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 12:34:50.198552 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:34:50.200380 28203 raft_consensus.cc:1261] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074 [term 1 FOLLOWER]: Refusing update from remote peer 54a521a23d8342258547396d5c05c636: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:50.200397 28072 raft_consensus.cc:1261] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580 [term 1 FOLLOWER]: Refusing update from remote peer 54a521a23d8342258547396d5c05c636: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:34:50.200723 28404 consensus_queue.cc:1035] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24b15dd270b42fbb434cfd8459d8074" member_type: VOTER last_known_addr { host: "127.8.13.2" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:50.200845 28401 consensus_queue.cc:1035] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a755f46f2cbf4fbe800129450b16b580" member_type: VOTER last_known_addr { host: "127.8.13.1" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:34:50.203912 28408 mvcc.cc:205] Tried to move back new op lower bound from 7334293054258364416 to 7334293054197866496. Current Snapshot: MvccSnapshot[applied={T|T < 7334293054258364416}]
I20260928 12:34:50.205262 28409 mvcc.cc:205] Tried to move back new op lower bound from 7334293054258364416 to 7334293054197866496. Current Snapshot: MvccSnapshot[applied={T|T < 7334293054258364416 or (T in {7334293054258364416})}]
I20260928 12:34:50.205519 27929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:50.205619 27929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:34:50.206409 27929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64f341aa4b5c4ef6bfca265b052db06d: Sending DeleteTablet for 3 replicas of tablet a9004e31fd5f47aba6056967135cb4f9
I20260928 12:34:50.206710 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet a9004e31fd5f47aba6056967135cb4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:50 UTC) from {username='slave'} at 127.0.0.1:50864
I20260928 12:34:50.206710 28314 tablet_service.cc:1514] Processing DeleteTablet for tablet a9004e31fd5f47aba6056967135cb4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:50 UTC) from {username='slave'} at 127.0.0.1:57924
I20260928 12:34:50.207103 28417 tablet_replica.cc:330] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074: stopping tablet replica
I20260928 12:34:50.207093 28052 tablet_service.cc:1514] Processing DeleteTablet for tablet a9004e31fd5f47aba6056967135cb4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:34:50 UTC) from {username='slave'} at 127.0.0.1:46386
I20260928 12:34:50.207232 28417 raft_consensus.cc:2229] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:34:50.207338 28417 raft_consensus.cc:2258] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:50.207391 28418 tablet_replica.cc:330] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580: stopping tablet replica
I20260928 12:34:50.207470 28418 raft_consensus.cc:2229] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:34:50.207561 28418 raft_consensus.cc:2258] T a9004e31fd5f47aba6056967135cb4f9 P a755f46f2cbf4fbe800129450b16b580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:50.207544 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 27990
I20260928 12:34:50.208537 28417 ts_tablet_manager.cc:1905] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:50.209920 28416 tablet_replica.cc:330] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636: stopping tablet replica
I20260928 12:34:50.210103 28416 raft_consensus.cc:2229] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:34:50.210258 28416 raft_consensus.cc:2258] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:34:50.210367 28417 ts_tablet_manager.cc:1918] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:50.210434 28417 log.cc:1196] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-1/wal/wals/a9004e31fd5f47aba6056967135cb4f9
I20260928 12:34:50.210803 28416 ts_tablet_manager.cc:1905] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:34:50.210840 28417 ts_tablet_manager.cc:1939] T a9004e31fd5f47aba6056967135cb4f9 P a24b15dd270b42fbb434cfd8459d8074: Deleting consensus metadata
I20260928 12:34:50.211220 27914 catalog_manager.cc:4841] TS a24b15dd270b42fbb434cfd8459d8074 (127.8.13.2:34093): tablet a9004e31fd5f47aba6056967135cb4f9 (table test-table [id=ef6606f542de4b6caaf85329bb742655]) successfully deleted
I20260928 12:34:50.212528 28416 ts_tablet_manager.cc:1918] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:34:50.212603 28416 log.cc:1196] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636: Deleting WAL directory at /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598640018661-8244-0/minicluster-data/ts-2/wal/wals/a9004e31fd5f47aba6056967135cb4f9
I20260928 12:34:50.212836 28416 ts_tablet_manager.cc:1939] T a9004e31fd5f47aba6056967135cb4f9 P 54a521a23d8342258547396d5c05c636: Deleting consensus metadata
I20260928 12:34:50.213248 27916 catalog_manager.cc:4841] TS 54a521a23d8342258547396d5c05c636 (127.8.13.3:42215): tablet a9004e31fd5f47aba6056967135cb4f9 (table test-table [id=ef6606f542de4b6caaf85329bb742655]) successfully deleted
W20260928 12:34:50.213408 27915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.1:40523 (error 108)
I20260928 12:34:50.213892 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:50.213743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.1:40523 (local address 127.0.0.1:46396)
0928 12:34:50.213817 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 12:34:50.213860 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.1:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 12:34:50.213901 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 28121
W20260928 12:34:50.218276 27915 catalog_manager.cc:4584] TS a755f46f2cbf4fbe800129450b16b580 (127.8.13.1:40523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9004e31fd5f47aba6056967135cb4f9: Network error: Client connection negotiation failed: client connection to 127.8.13.1:40523: connect: Connection refused (error 111)
I20260928 12:34:50.219188 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 28252
I20260928 12:34:50.223991 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 27899
2026-09-28T12:34:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (716 ms)
[----------] 6 tests from EncryptionPolicyTest (8117 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:50 dist-test-slave-g31s krb5kdc[28422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:50 dist-test-slave-g31s krb5kdc[28422](info): set up 2 sockets
Sep 28 12:34:50 dist-test-slave-g31s krb5kdc[28422](info): commencing operation
krb5kdc: starting...
W20260928 12:34:51.165653 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:34:51 dist-test-slave-g31s krb5kdc[28422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:34:51.280483 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:51.361613 28435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:51.361886 28435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:51.361912 28435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:51.363268 28435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:51.363338 28435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:51.363351 28435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:51.363370 28435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:51.363381 28435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:51.365161 28435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:51.365449 28435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:51.365693 28435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:51.370689 28440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:51.370705 28441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:51.370689 28443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:51.371208 28435 server_base.cc:1034] running on GCE node
I20260928 12:34:51.371456 28435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:51.371758 28435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:51.372942 28435 hybrid_clock.cc:648] HybridClock initialized: now 1790598891372922 us; error 32 us; skew 500 ppm
Sep 28 12:34:51 dist-test-slave-g31s krb5kdc[28422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:51.375547 28435 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:34:51.376062 28435 webserver.cc:460] Webserver started at http://127.0.0.1:35091/ using document root <none> and password file <none>
I20260928 12:34:51.376309 28435 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:51.376360 28435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:51.376475 28435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:51.377592 28435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "94f5db02fad64345b502d1915b794d88"
format_stamp: "Formatted at 2026-09-28 12:34:51 on dist-test-slave-g31s"
server_key: "79473e09fffe870ce2888298f5dadce1"
server_key_iv: "f8818c3371c609c6dfa8cf9ae26c221a"
server_key_version: "encryptionkey@0"
I20260928 12:34:51.377975 28435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "94f5db02fad64345b502d1915b794d88"
format_stamp: "Formatted at 2026-09-28 12:34:51 on dist-test-slave-g31s"
server_key: "79473e09fffe870ce2888298f5dadce1"
server_key_iv: "f8818c3371c609c6dfa8cf9ae26c221a"
server_key_version: "encryptionkey@0"
I20260928 12:34:51.379289 28435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:34:51.380090 28450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:51.380247 28435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:34:51.380330 28435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "94f5db02fad64345b502d1915b794d88"
format_stamp: "Formatted at 2026-09-28 12:34:51 on dist-test-slave-g31s"
server_key: "79473e09fffe870ce2888298f5dadce1"
server_key_iv: "f8818c3371c609c6dfa8cf9ae26c221a"
server_key_version: "encryptionkey@0"
I20260928 12:34:51.380404 28435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:51.393602 28435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:51.395754 28435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:51.395871 28435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:51.399824 28435 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45045
I20260928 12:34:51.399868 28502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45045 every 8 connection(s)
I20260928 12:34:51.400209 28435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:51.401167 28503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:51.403458 28503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88: Bootstrap starting.
I20260928 12:34:51.403980 28503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:51.404210 28503 log.cc:824] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:51.404824 28503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88: No bootstrap required, opened a new log
I20260928 12:34:51.405280 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 28435
I20260928 12:34:51.405377 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:51.405547 8244 external_mini_cluster.cc:1428] Setting key 536d1423d5d4ad26c8a2a8b2dff0f6cb
I20260928 12:34:51.406633 28503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f5db02fad64345b502d1915b794d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 12:34:51.406777 28503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:51.406811 28503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f5db02fad64345b502d1915b794d88, State: Initialized, Role: FOLLOWER
I20260928 12:34:51.406944 28503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f5db02fad64345b502d1915b794d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 12:34:51.407020 28503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:51.407056 28503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:51.407101 28503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:51.408017 28503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f5db02fad64345b502d1915b794d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 12:34:51.408113 28503 leader_election.cc:302] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94f5db02fad64345b502d1915b794d88; no voters:
I20260928 12:34:51.408250 28503 leader_election.cc:288] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:51.408282 28508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:51.408412 28508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [term 1 LEADER]: Becoming Leader. State: Replica: 94f5db02fad64345b502d1915b794d88, State: Running, Role: LEADER
I20260928 12:34:51.408486 28503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:51.408483 28508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f5db02fad64345b502d1915b794d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 12:34:51.410301 28510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94f5db02fad64345b502d1915b794d88. Latest consensus state: current_term: 1 leader_uuid: "94f5db02fad64345b502d1915b794d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f5db02fad64345b502d1915b794d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } } }
I20260928 12:34:51.410427 28509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94f5db02fad64345b502d1915b794d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f5db02fad64345b502d1915b794d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } } }
I20260928 12:34:51.410847 28510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:51.410884 28509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88 [sys.catalog]: This master's current role is: LEADER
W20260928 12:34:51.411147 28522 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:34:51.411206 28522 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 12:34:51 dist-test-slave-g31s krb5kdc[28422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:34:51.412526 28524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:51.412722 28524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:51.415323 28524 catalog_manager.cc:1347] Generated new cluster ID: fa738cfc77004f5e98e5c937137e8c8f
I20260928 12:34:51.415377 28524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:51.416437 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:51.406108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53575 (local address 127.0.0.1:45045)
0928 12:34:51.406266 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:51.406275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:51.406305 (+ 30us) server_negotiation.cc:407] Connection header received
0928 12:34:51.406603 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:51.406608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:51.406652 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:51.406757 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:51.407511 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:51.408815 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:51.409596 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:51.410168 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:51.410275 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:51.412923 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:51.412927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:51.412927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:51.412939 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:51.414818 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:51.415175 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:51.415178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:51.415179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:51.415293 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:51.415551 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:51.415552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:51.415552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:51.415877 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:51.416043 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:51.416124 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 12:34:51.416223 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260928 12:34:51.417783 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:34:51.419531 28524 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:51.420186 28524 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:51.423676 28524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94f5db02fad64345b502d1915b794d88: Generated new TSK 0
I20260928 12:34:51.423831 28524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:34:51.432232 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:51.422048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51115 (local address 127.0.0.1:45045)
0928 12:34:51.422083 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:51.422085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:51.425979 (+ 3894us) server_negotiation.cc:407] Connection header received
0928 12:34:51.426040 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:51.426040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:51.426050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:51.426099 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:51.426712 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:51.427215 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:51.428235 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:51.428445 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:51.428492 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:51.429834 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:51.429836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:51.429836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:51.429854 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:51.431123 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:51.431548 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:51.431549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:51.431549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:51.431610 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:51.431811 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:51.431811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:51.431811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:51.431909 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:51.431981 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:51.432120 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 12:34:51.432170 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:34:51.434075 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 28435
W20260928 12:34:51.438170 28420 connection.cc:535] client connection to 127.0.0.1:45045 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45045): Connection reset by peer (error 104)
2026-09-28T12:34:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:51 dist-test-slave-g31s krb5kdc[28534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:51 dist-test-slave-g31s krb5kdc[28534](info): set up 2 sockets
Sep 28 12:34:51 dist-test-slave-g31s krb5kdc[28534](info): commencing operation
krb5kdc: starting...
W20260928 12:34:52.385617 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:34:52 dist-test-slave-g31s krb5kdc[28534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:34:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:34:52.500767 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:52.581003 28547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:52.581228 28547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:52.581251 28547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:52.582448 28547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:52.582489 28547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:52.582501 28547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:52.582512 28547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:52.582523 28547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:52.584039 28547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:52.584260 28547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:52.584435 28547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:52.589438 28552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:52.589433 28553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:52.589545 28547 server_base.cc:1034] running on GCE node
W20260928 12:34:52.589445 28555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:52.589929 28547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:52.590224 28547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:52.591363 28547 hybrid_clock.cc:648] HybridClock initialized: now 1790598892591350 us; error 26 us; skew 500 ppm
Sep 28 12:34:52 dist-test-slave-g31s krb5kdc[28534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:52.598206 28547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:34:52.598690 28547 webserver.cc:460] Webserver started at http://127.0.0.1:46711/ using document root <none> and password file <none>
I20260928 12:34:52.598910 28547 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:52.598960 28547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:52.599092 28547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:52.600181 28547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "aed84e8cf6534bd9bcdca49ae0d59e7a"
format_stamp: "Formatted at 2026-09-28 12:34:52 on dist-test-slave-g31s"
server_key: "dd846113d86fc1e6a0532efa9eae3220"
server_key_iv: "e6761e26a20d4e07301443d814b49ea0"
server_key_version: "encryptionkey@0"
I20260928 12:34:52.600531 28547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "aed84e8cf6534bd9bcdca49ae0d59e7a"
format_stamp: "Formatted at 2026-09-28 12:34:52 on dist-test-slave-g31s"
server_key: "dd846113d86fc1e6a0532efa9eae3220"
server_key_iv: "e6761e26a20d4e07301443d814b49ea0"
server_key_version: "encryptionkey@0"
I20260928 12:34:52.601897 28547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:34:52.602751 28562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:52.602932 28547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:52.603027 28547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "aed84e8cf6534bd9bcdca49ae0d59e7a"
format_stamp: "Formatted at 2026-09-28 12:34:52 on dist-test-slave-g31s"
server_key: "dd846113d86fc1e6a0532efa9eae3220"
server_key_iv: "e6761e26a20d4e07301443d814b49ea0"
server_key_version: "encryptionkey@0"
I20260928 12:34:52.603102 28547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:52.628345 28547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:52.630473 28547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:52.630610 28547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:52.634294 28547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41855
I20260928 12:34:52.634330 28614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41855 every 8 connection(s)
I20260928 12:34:52.634675 28547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:52.635550 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 28547
I20260928 12:34:52.635659 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:52.635694 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:52.635866 8244 external_mini_cluster.cc:1428] Setting key f7ae4b39f245ebcc8a7904d0b484180a
I20260928 12:34:52.638820 28615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a: Bootstrap starting.
I20260928 12:34:52.639590 28615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:52.639858 28615 log.cc:824] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:52.640595 28615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a: No bootstrap required, opened a new log
I20260928 12:34:52.641352 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:52.636388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53947 (local address 127.0.0.1:41855)
0928 12:34:52.636552 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:52.636560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:52.636586 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:34:52.637127 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:52.637134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:52.637184 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:52.637306 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:52.637529 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:52.638473 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:52.639511 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:52.639983 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:52.640079 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:52.640741 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:52.640746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:52.640747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:52.640912 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:52.640945 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:52.640957 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:52.641026 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:52.641136 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 12:34:52.641166 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260928 12:34:52.642359 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:34:52.642699 28615 raft_consensus.cc:348] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed84e8cf6534bd9bcdca49ae0d59e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } }
I20260928 12:34:52.642844 28615 raft_consensus.cc:374] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:52.642876 28615 raft_consensus.cc:729] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed84e8cf6534bd9bcdca49ae0d59e7a, State: Initialized, Role: FOLLOWER
I20260928 12:34:52.643000 28615 consensus_queue.cc:260] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed84e8cf6534bd9bcdca49ae0d59e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } }
I20260928 12:34:52.643072 28615 raft_consensus.cc:388] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:52.643102 28615 raft_consensus.cc:482] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:52.643149 28615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:52.643887 28615 raft_consensus.cc:504] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed84e8cf6534bd9bcdca49ae0d59e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } }
I20260928 12:34:52.643992 28615 leader_election.cc:302] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aed84e8cf6534bd9bcdca49ae0d59e7a; no voters:
I20260928 12:34:52.644177 28615 leader_election.cc:288] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:52.644275 28625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:52.644423 28625 raft_consensus.cc:686] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [term 1 LEADER]: Becoming Leader. State: Replica: aed84e8cf6534bd9bcdca49ae0d59e7a, State: Running, Role: LEADER
I20260928 12:34:52.644510 28615 sys_catalog.cc:564] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:52.644579 28625 consensus_queue.cc:237] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed84e8cf6534bd9bcdca49ae0d59e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } }
I20260928 12:34:52.644976 28625 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aed84e8cf6534bd9bcdca49ae0d59e7a. Latest consensus state: current_term: 1 leader_uuid: "aed84e8cf6534bd9bcdca49ae0d59e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed84e8cf6534bd9bcdca49ae0d59e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } } }
I20260928 12:34:52.645185 28625 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:52.645123 28626 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aed84e8cf6534bd9bcdca49ae0d59e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed84e8cf6534bd9bcdca49ae0d59e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } } }
I20260928 12:34:52.645560 28626 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a [sys.catalog]: This master's current role is: LEADER
W20260928 12:34:52.646194 28641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:34:52.646248 28641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:34:52.646582 28634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:52.646778 28634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:52.647050 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:52.643701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47691 (local address 127.0.0.1:41855)
0928 12:34:52.643736 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:52.643738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:52.643837 (+ 99us) server_negotiation.cc:407] Connection header received
0928 12:34:52.643900 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:52.643901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:52.643909 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:52.643961 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:52.644089 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:52.644705 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:52.645606 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:52.645910 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:52.645973 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:52.646680 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:52.646682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:52.646682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:52.646687 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:52.646701 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:52.646705 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:52.646749 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:52.646973 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 12:34:52.646984 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:34:52.648342 28634 catalog_manager.cc:1347] Generated new cluster ID: 973cd8b793a745e49ed01a9ead3e7648
I20260928 12:34:52.648391 28634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:52.653138 28634 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:52.653719 28634 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:52.656843 28634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aed84e8cf6534bd9bcdca49ae0d59e7a: Generated new TSK 0
I20260928 12:34:52.657037 28634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:34:52.664834 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 28547
W20260928 12:34:52.668846 28532 connection.cc:535] client connection to 127.0.0.1:41855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41855): Connection reset by peer (error 104)
2026-09-28T12:34:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:52 dist-test-slave-g31s krb5kdc[28646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:52 dist-test-slave-g31s krb5kdc[28646](info): set up 2 sockets
Sep 28 12:34:52 dist-test-slave-g31s krb5kdc[28646](info): commencing operation
krb5kdc: starting...
W20260928 12:34:53.613791 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:34:53 dist-test-slave-g31s krb5kdc[28646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:34:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:34:53.726593 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:53.806854 28659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:53.807066 28659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:53.807089 28659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:53.808254 28659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:53.808300 28659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:53.808313 28659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:53.808326 28659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:53.808336 28659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:53.809825 28659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:53.810058 28659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:53.810268 28659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:53.814742 28667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:53.814808 28659 server_base.cc:1034] running on GCE node
W20260928 12:34:53.814754 28664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:53.814757 28665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:53.815120 28659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:53.815398 28659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:53.816547 28659 hybrid_clock.cc:648] HybridClock initialized: now 1790598893816531 us; error 28 us; skew 500 ppm
Sep 28 12:34:53 dist-test-slave-g31s krb5kdc[28646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:53.818811 28659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:34:53.819192 28659 webserver.cc:460] Webserver started at http://127.0.0.1:37327/ using document root <none> and password file <none>
I20260928 12:34:53.819396 28659 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:53.819448 28659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:53.819566 28659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:53.820619 28659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "a39bf49b44ff4d0bb0adefb7639f5d71"
format_stamp: "Formatted at 2026-09-28 12:34:53 on dist-test-slave-g31s"
server_key: "9849ea13df21bd314d684706cadcc679"
server_key_iv: "e3d69dfe68ca63b2a27869ccc3df65b8"
server_key_version: "encryptionkey@0"
I20260928 12:34:53.820925 28659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "a39bf49b44ff4d0bb0adefb7639f5d71"
format_stamp: "Formatted at 2026-09-28 12:34:53 on dist-test-slave-g31s"
server_key: "9849ea13df21bd314d684706cadcc679"
server_key_iv: "e3d69dfe68ca63b2a27869ccc3df65b8"
server_key_version: "encryptionkey@0"
I20260928 12:34:53.822137 28659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:34:53.822747 28674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:53.822889 28659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:34:53.822978 28659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "a39bf49b44ff4d0bb0adefb7639f5d71"
format_stamp: "Formatted at 2026-09-28 12:34:53 on dist-test-slave-g31s"
server_key: "9849ea13df21bd314d684706cadcc679"
server_key_iv: "e3d69dfe68ca63b2a27869ccc3df65b8"
server_key_version: "encryptionkey@0"
I20260928 12:34:53.823031 28659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:53.826119 28659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:53.827782 28659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:53.827909 28659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:53.831586 28659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38077
I20260928 12:34:53.831616 28726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38077 every 8 connection(s)
I20260928 12:34:53.831897 28659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:53.832764 28727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:53.835000 28727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71: Bootstrap starting.
I20260928 12:34:53.835561 28727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:53.835813 28727 log.cc:824] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:53.836501 28727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71: No bootstrap required, opened a new log
I20260928 12:34:53.838162 28727 raft_consensus.cc:348] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39bf49b44ff4d0bb0adefb7639f5d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260928 12:34:53.838311 28727 raft_consensus.cc:374] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:53.838332 28727 raft_consensus.cc:729] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39bf49b44ff4d0bb0adefb7639f5d71, State: Initialized, Role: FOLLOWER
I20260928 12:34:53.838477 28727 consensus_queue.cc:260] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39bf49b44ff4d0bb0adefb7639f5d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260928 12:34:53.838539 28727 raft_consensus.cc:388] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:53.838562 28727 raft_consensus.cc:482] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:53.838604 28727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:53.839150 28727 raft_consensus.cc:504] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39bf49b44ff4d0bb0adefb7639f5d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260928 12:34:53.839243 28727 leader_election.cc:302] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a39bf49b44ff4d0bb0adefb7639f5d71; no voters:
I20260928 12:34:53.839412 28727 leader_election.cc:288] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:53.839457 28730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:53.839593 28730 raft_consensus.cc:686] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [term 1 LEADER]: Becoming Leader. State: Replica: a39bf49b44ff4d0bb0adefb7639f5d71, State: Running, Role: LEADER
I20260928 12:34:53.839679 28727 sys_catalog.cc:564] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:53.839685 28730 consensus_queue.cc:237] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39bf49b44ff4d0bb0adefb7639f5d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260928 12:34:53.840078 28732 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a39bf49b44ff4d0bb0adefb7639f5d71. Latest consensus state: current_term: 1 leader_uuid: "a39bf49b44ff4d0bb0adefb7639f5d71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39bf49b44ff4d0bb0adefb7639f5d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } } }
I20260928 12:34:53.840214 28731 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a39bf49b44ff4d0bb0adefb7639f5d71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39bf49b44ff4d0bb0adefb7639f5d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } } }
I20260928 12:34:53.840212 28732 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:53.840317 28731 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:53.840797 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 28659
I20260928 12:34:53.840901 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:53.841074 8244 external_mini_cluster.cc:1428] Setting key b263c039f50b971b67426d2ce0f6ec53
I20260928 12:34:53.841312 28739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:53.841912 28739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:53.843387 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:53.841698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50863 (local address 127.0.0.1:38077)
0928 12:34:53.841988 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:53.841998 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:53.842031 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:34:53.842161 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:53.842168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:53.842199 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:53.842341 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:53.842962 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:53.842965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:53.842966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:53.842970 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:53.843007 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:53.843022 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:53.843088 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:53.843163 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 12:34:53.843205 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":73,"threads_started":1}
I20260928 12:34:53.843621 28739 catalog_manager.cc:1347] Generated new cluster ID: 01dad8781d3a4d5cb299b8fa9ba5c6ff
I20260928 12:34:53.843667 28739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:53.846552 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:34:53.849251 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:53.848173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58103 (local address 127.0.0.1:38077)
0928 12:34:53.848218 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:53.848220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:53.848305 (+ 85us) server_negotiation.cc:407] Connection header received
0928 12:34:53.848374 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:53.848375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:53.848382 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:53.848425 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:53.849025 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:53.849026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:53.849027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:53.849030 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:53.849046 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:53.849051 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:53.849100 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:53.849156 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 12:34:53.849169 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:34:53.865094 28739 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:53.865293 28739 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:53.873982 28739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a39bf49b44ff4d0bb0adefb7639f5d71: Generated new TSK 0
I20260928 12:34:53.874225 28739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:34:53.876808 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 28659
2026-09-28T12:34:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:53 dist-test-slave-g31s krb5kdc[28758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:53 dist-test-slave-g31s krb5kdc[28758](info): set up 2 sockets
Sep 28 12:34:53 dist-test-slave-g31s krb5kdc[28758](info): commencing operation
krb5kdc: starting...
W20260928 12:34:54.837551 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:34:54 dist-test-slave-g31s krb5kdc[28758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:34:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:34:54.950336 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:55.031239 28771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:55.031448 28771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:55.031481 28771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:55.032677 28771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:55.032727 28771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:55.032745 28771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:55.032761 28771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:55.032776 28771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:55.034320 28771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:55.034575 28771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:55.034778 28771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:55.039405 28776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:55.039598 28771 server_base.cc:1034] running on GCE node
W20260928 12:34:55.039419 28777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:55.039413 28779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:55.039976 28771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:55.040266 28771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:55.041404 28771 hybrid_clock.cc:648] HybridClock initialized: now 1790598895041390 us; error 29 us; skew 500 ppm
Sep 28 12:34:55 dist-test-slave-g31s krb5kdc[28758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:55.044091 28771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:34:55.044559 28771 webserver.cc:460] Webserver started at http://127.0.0.1:41007/ using document root <none> and password file <none>
I20260928 12:34:55.044765 28771 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:55.044813 28771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:55.044947 28771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:55.046061 28771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "718ffd5ad86a4d8e8130d3195cbb2023"
format_stamp: "Formatted at 2026-09-28 12:34:55 on dist-test-slave-g31s"
server_key: "cb1048d8bf878a7fca5454bbfe069322"
server_key_iv: "b0a36267415a1b6498aa3088fbf9d180"
server_key_version: "encryptionkey@0"
I20260928 12:34:55.046382 28771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "718ffd5ad86a4d8e8130d3195cbb2023"
format_stamp: "Formatted at 2026-09-28 12:34:55 on dist-test-slave-g31s"
server_key: "cb1048d8bf878a7fca5454bbfe069322"
server_key_iv: "b0a36267415a1b6498aa3088fbf9d180"
server_key_version: "encryptionkey@0"
I20260928 12:34:55.047613 28771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:55.048369 28786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:55.048585 28771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:34:55.048681 28771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "718ffd5ad86a4d8e8130d3195cbb2023"
format_stamp: "Formatted at 2026-09-28 12:34:55 on dist-test-slave-g31s"
server_key: "cb1048d8bf878a7fca5454bbfe069322"
server_key_iv: "b0a36267415a1b6498aa3088fbf9d180"
server_key_version: "encryptionkey@0"
I20260928 12:34:55.048743 28771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:55.061456 28771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:55.063377 28771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:55.063529 28771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:55.067440 28771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46703
I20260928 12:34:55.067471 28838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46703 every 8 connection(s)
I20260928 12:34:55.067799 28771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:55.068745 28839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:55.071098 28839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023: Bootstrap starting.
I20260928 12:34:55.071640 28839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:55.071861 28839 log.cc:824] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:55.072499 28839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023: No bootstrap required, opened a new log
I20260928 12:34:55.074146 28839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718ffd5ad86a4d8e8130d3195cbb2023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46703 } }
I20260928 12:34:55.074290 28839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:55.074311 28839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718ffd5ad86a4d8e8130d3195cbb2023, State: Initialized, Role: FOLLOWER
I20260928 12:34:55.074416 28839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718ffd5ad86a4d8e8130d3195cbb2023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46703 } }
I20260928 12:34:55.074467 28839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:55.074486 28839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:55.074512 28839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:55.075011 28839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718ffd5ad86a4d8e8130d3195cbb2023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46703 } }
I20260928 12:34:55.075131 28839 leader_election.cc:302] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 718ffd5ad86a4d8e8130d3195cbb2023; no voters:
I20260928 12:34:55.075158 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 28771
I20260928 12:34:55.075264 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:55.075311 28839 leader_election.cc:288] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:55.075361 28842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:55.075474 8244 external_mini_cluster.cc:1428] Setting key e13a62f295ada055e07e7e91d42cb908
I20260928 12:34:55.075521 28842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [term 1 LEADER]: Becoming Leader. State: Replica: 718ffd5ad86a4d8e8130d3195cbb2023, State: Running, Role: LEADER
I20260928 12:34:55.075582 28839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:55.075654 28842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718ffd5ad86a4d8e8130d3195cbb2023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46703 } }
I20260928 12:34:55.076007 28842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 718ffd5ad86a4d8e8130d3195cbb2023. Latest consensus state: current_term: 1 leader_uuid: "718ffd5ad86a4d8e8130d3195cbb2023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718ffd5ad86a4d8e8130d3195cbb2023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46703 } } }
I20260928 12:34:55.076138 28842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:55.076174 28844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "718ffd5ad86a4d8e8130d3195cbb2023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718ffd5ad86a4d8e8130d3195cbb2023" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46703 } } }
I20260928 12:34:55.076299 28844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:55.076367 28848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:55.076917 28848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:55.078624 28848 catalog_manager.cc:1347] Generated new cluster ID: d471e1fd8c22475abbc06168b2ccd47b
I20260928 12:34:55.078675 28848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:34:55 dist-test-slave-g31s krb5kdc[28758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:34:55.086329 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:55.076024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35249 (local address 127.0.0.1:46703)
0928 12:34:55.076391 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:55.076401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:55.076440 (+ 39us) server_negotiation.cc:407] Connection header received
0928 12:34:55.076584 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:55.076594 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:55.076658 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:55.076801 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:55.077621 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:55.079022 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:55.079782 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:55.080211 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:55.080306 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:55.082594 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:55.082599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:55.082599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:55.082611 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:55.084775 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:55.085124 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:55.085127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:55.085128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:55.085230 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:55.085409 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:55.085410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:55.085410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:55.085678 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:55.085866 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:55.085966 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 12:34:55.086111 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":74,"threads_started":1}
I20260928 12:34:55.087272 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:34:55.091704 28848 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:55.092202 28848 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:55.094877 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:55.088539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38665 (local address 127.0.0.1:46703)
0928 12:34:55.088581 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:55.088582 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:55.088686 (+ 104us) server_negotiation.cc:407] Connection header received
0928 12:34:55.088740 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:55.088741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:55.088751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:55.088795 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:55.089282 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:55.089898 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:55.090582 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:55.090843 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:55.090896 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:55.092261 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:55.092263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:55.092263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:55.092270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:55.093750 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:55.094145 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:55.094146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:55.094147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:55.094200 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:55.094373 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:55.094374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:55.094374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:55.094495 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:55.094591 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:55.094690 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 12:34:55.094780 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:34:55.103067 28848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 718ffd5ad86a4d8e8130d3195cbb2023: Generated new TSK 0
I20260928 12:34:55.103296 28848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:34:55.106402 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 28771
2026-09-28T12:34:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:55 dist-test-slave-g31s krb5kdc[28870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:55 dist-test-slave-g31s krb5kdc[28870](info): set up 2 sockets
Sep 28 12:34:55 dist-test-slave-g31s krb5kdc[28870](info): commencing operation
krb5kdc: starting...
W20260928 12:34:56.045574 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:34:56 dist-test-slave-g31s krb5kdc[28870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:34:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:34:56.158963 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:56.241542 28883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:56.241816 28883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:56.241844 28883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:56.243124 28883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:56.243180 28883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:56.243193 28883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:56.243206 28883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:56.243217 28883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:56.244810 28883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:56.245044 28883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:56.245227 28883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:34:56.250186 28883 server_base.cc:1034] running on GCE node
W20260928 12:34:56.250134 28889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:56.250146 28888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:56.250162 28891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:56.250607 28883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:56.250941 28883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:56.252089 28883 hybrid_clock.cc:648] HybridClock initialized: now 1790598896252067 us; error 35 us; skew 500 ppm
Sep 28 12:34:56 dist-test-slave-g31s krb5kdc[28870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:56.254644 28883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:34:56.255128 28883 webserver.cc:460] Webserver started at http://127.0.0.1:42057/ using document root <none> and password file <none>
I20260928 12:34:56.255342 28883 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:56.255398 28883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:56.255533 28883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:56.256698 28883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "11603e797b8447fb8d5b5fa1dd007c61"
format_stamp: "Formatted at 2026-09-28 12:34:56 on dist-test-slave-g31s"
server_key: "978b203ada2ef019d1aeb32fa89a3b72"
server_key_iv: "d0b09fc15c77bfa6928cc9782ee8bea6"
server_key_version: "encryptionkey@0"
I20260928 12:34:56.257035 28883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "11603e797b8447fb8d5b5fa1dd007c61"
format_stamp: "Formatted at 2026-09-28 12:34:56 on dist-test-slave-g31s"
server_key: "978b203ada2ef019d1aeb32fa89a3b72"
server_key_iv: "d0b09fc15c77bfa6928cc9782ee8bea6"
server_key_version: "encryptionkey@0"
I20260928 12:34:56.258433 28883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:34:56.259141 28898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:56.259279 28883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:56.259364 28883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "11603e797b8447fb8d5b5fa1dd007c61"
format_stamp: "Formatted at 2026-09-28 12:34:56 on dist-test-slave-g31s"
server_key: "978b203ada2ef019d1aeb32fa89a3b72"
server_key_iv: "d0b09fc15c77bfa6928cc9782ee8bea6"
server_key_version: "encryptionkey@0"
I20260928 12:34:56.259433 28883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:56.280725 28883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:56.282655 28883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:56.282799 28883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:56.286553 28883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41579
I20260928 12:34:56.286592 28950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41579 every 8 connection(s)
I20260928 12:34:56.286917 28883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:56.287907 28951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:56.290365 28951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61: Bootstrap starting.
I20260928 12:34:56.290969 28951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:56.291234 28951 log.cc:824] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:56.291944 28951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61: No bootstrap required, opened a new log
I20260928 12:34:56.293601 28951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11603e797b8447fb8d5b5fa1dd007c61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260928 12:34:56.293761 28951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:56.293783 28951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11603e797b8447fb8d5b5fa1dd007c61, State: Initialized, Role: FOLLOWER
I20260928 12:34:56.293964 28951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11603e797b8447fb8d5b5fa1dd007c61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260928 12:34:56.294034 28951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:56.294055 28951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:56.294102 28951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:56.294242 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 28883
I20260928 12:34:56.294334 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:56.294471 8244 external_mini_cluster.cc:1428] Setting key bda10a10f004da33fb84990582b01158
I20260928 12:34:56.294687 28951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11603e797b8447fb8d5b5fa1dd007c61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260928 12:34:56.294806 28951 leader_election.cc:302] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11603e797b8447fb8d5b5fa1dd007c61; no voters:
I20260928 12:34:56.294996 28951 leader_election.cc:288] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:56.295171 28956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:56.295292 28951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:56.295360 28956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [term 1 LEADER]: Becoming Leader. State: Replica: 11603e797b8447fb8d5b5fa1dd007c61, State: Running, Role: LEADER
I20260928 12:34:56.295475 28956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11603e797b8447fb8d5b5fa1dd007c61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260928 12:34:56.295809 28956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11603e797b8447fb8d5b5fa1dd007c61. Latest consensus state: current_term: 1 leader_uuid: "11603e797b8447fb8d5b5fa1dd007c61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11603e797b8447fb8d5b5fa1dd007c61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } } }
I20260928 12:34:56.295857 28954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11603e797b8447fb8d5b5fa1dd007c61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11603e797b8447fb8d5b5fa1dd007c61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } } }
I20260928 12:34:56.295936 28956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:56.295994 28954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:56.296193 28962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:56.296909 28962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:56.298627 28962 catalog_manager.cc:1347] Generated new cluster ID: 2fd2de7dfdb143c4b4f6e786b791aa8e
I20260928 12:34:56.298684 28962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:56.300434 28957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:56.295098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51661 (local address 127.0.0.1:41579)
0928 12:34:56.295259 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:56.295269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:56.295303 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:34:56.295438 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:56.295446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:56.295509 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:56.295654 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:56.296038 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:56.297494 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:56.298545 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:56.299015 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:56.299076 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:56.299808 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:56.299813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:56.299813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:56.299994 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:56.300024 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:56.300037 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:56.300091 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:56.300173 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 12:34:56.300258 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260928 12:34:56.301308 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:34:56.308879 28957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:56.306138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50841 (local address 127.0.0.1:41579)
0928 12:34:56.306180 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:56.306183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:56.306381 (+ 198us) server_negotiation.cc:407] Connection header received
0928 12:34:56.306438 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:56.306439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:56.306462 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:56.306508 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:56.306638 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:56.307133 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:56.307815 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:56.308025 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:56.308078 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:56.308655 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:56.308656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:56.308656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:56.308661 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:56.308676 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:56.308680 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:56.308724 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:56.308772 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 12:34:56.308798 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:34:56.312372 28962 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:56.313007 28962 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:56.316558 28962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11603e797b8447fb8d5b5fa1dd007c61: Generated new TSK 0
I20260928 12:34:56.316711 28962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:34:56.319706 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 28883
2026-09-28T12:34:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:56 dist-test-slave-g31s krb5kdc[28982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:56 dist-test-slave-g31s krb5kdc[28982](info): set up 2 sockets
Sep 28 12:34:56 dist-test-slave-g31s krb5kdc[28982](info): commencing operation
krb5kdc: starting...
W20260928 12:34:57.261662 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:34:57 dist-test-slave-g31s krb5kdc[28982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:34:57.374328 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:57.453152 28995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:57.453385 28995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:57.453419 28995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:57.454697 28995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:57.454754 28995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:57.454773 28995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:57.454789 28995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:57.454804 28995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:57.456341 28995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:57.456595 28995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:57.456805 28995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:34:57.461547 28995 server_base.cc:1034] running on GCE node
W20260928 12:34:57.461490 29001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:57.461498 29000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:57.461515 29003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:57.461970 28995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:57.462229 28995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:57.463369 28995 hybrid_clock.cc:648] HybridClock initialized: now 1790598897463353 us; error 34 us; skew 500 ppm
Sep 28 12:34:57 dist-test-slave-g31s krb5kdc[28982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:57.465709 28995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:34:57.466161 28995 webserver.cc:460] Webserver started at http://127.0.0.1:45835/ using document root <none> and password file <none>
I20260928 12:34:57.466362 28995 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:57.466410 28995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:57.466544 28995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:57.467572 28995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "04d56cb4b8c4440d94fe0ce5a306f51f"
format_stamp: "Formatted at 2026-09-28 12:34:57 on dist-test-slave-g31s"
server_key: "23781b1cd1c3bdf2bdc634762dfaec92"
server_key_iv: "b30f1df1e406e4cf8d2ab8e0b2977f47"
server_key_version: "encryptionkey@0"
I20260928 12:34:57.467888 28995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "04d56cb4b8c4440d94fe0ce5a306f51f"
format_stamp: "Formatted at 2026-09-28 12:34:57 on dist-test-slave-g31s"
server_key: "23781b1cd1c3bdf2bdc634762dfaec92"
server_key_iv: "b30f1df1e406e4cf8d2ab8e0b2977f47"
server_key_version: "encryptionkey@0"
I20260928 12:34:57.469133 28995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:34:57.469782 29010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:57.469944 28995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:34:57.470022 28995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "04d56cb4b8c4440d94fe0ce5a306f51f"
format_stamp: "Formatted at 2026-09-28 12:34:57 on dist-test-slave-g31s"
server_key: "23781b1cd1c3bdf2bdc634762dfaec92"
server_key_iv: "b30f1df1e406e4cf8d2ab8e0b2977f47"
server_key_version: "encryptionkey@0"
I20260928 12:34:57.470081 28995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:57.473246 28995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:57.475455 28995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:57.475596 28995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:57.479311 28995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39797
I20260928 12:34:57.479342 29062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39797 every 8 connection(s)
I20260928 12:34:57.479661 28995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:57.480581 29063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:57.482923 29063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f: Bootstrap starting.
I20260928 12:34:57.483482 29063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:57.483742 29063 log.cc:824] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:57.484405 29063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f: No bootstrap required, opened a new log
I20260928 12:34:57.486119 29063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d56cb4b8c4440d94fe0ce5a306f51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } }
I20260928 12:34:57.486282 29063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:57.486304 29063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d56cb4b8c4440d94fe0ce5a306f51f, State: Initialized, Role: FOLLOWER
I20260928 12:34:57.486454 29063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d56cb4b8c4440d94fe0ce5a306f51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } }
I20260928 12:34:57.486519 29063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:57.486542 29063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:57.486600 29063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:57.487200 29063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d56cb4b8c4440d94fe0ce5a306f51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } }
I20260928 12:34:57.487301 29063 leader_election.cc:302] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04d56cb4b8c4440d94fe0ce5a306f51f; no voters:
I20260928 12:34:57.487485 29063 leader_election.cc:288] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:57.487529 29066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:57.487681 29066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [term 1 LEADER]: Becoming Leader. State: Replica: 04d56cb4b8c4440d94fe0ce5a306f51f, State: Running, Role: LEADER
I20260928 12:34:57.487762 29063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:57.487776 29066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d56cb4b8c4440d94fe0ce5a306f51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } }
I20260928 12:34:57.488159 29068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04d56cb4b8c4440d94fe0ce5a306f51f. Latest consensus state: current_term: 1 leader_uuid: "04d56cb4b8c4440d94fe0ce5a306f51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d56cb4b8c4440d94fe0ce5a306f51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } } }
I20260928 12:34:57.488237 29068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:57.488303 29067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04d56cb4b8c4440d94fe0ce5a306f51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d56cb4b8c4440d94fe0ce5a306f51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } } }
I20260928 12:34:57.488380 29067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:57.488701 29073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:57.488773 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 28995
I20260928 12:34:57.488884 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:57.489106 8244 external_mini_cluster.cc:1428] Setting key 09523136fbe997d897ec1e5c07d0c6b8
I20260928 12:34:57.489434 29073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:57.491688 29073 catalog_manager.cc:1347] Generated new cluster ID: ed871e9e71b54128ac12c49efac27b23
I20260928 12:34:57.491760 29073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:57.491859 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:57.489588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40161 (local address 127.0.0.1:39797)
0928 12:34:57.489923 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:57.489933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:57.489967 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:34:57.490209 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:57.490216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:57.490246 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:57.490381 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:57.491384 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:57.491388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:57.491389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:57.491394 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:57.491444 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:57.491456 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:57.491513 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:57.491577 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 12:34:57.491621 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":65,"threads_started":1}
I20260928 12:34:57.492712 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:34:57.497634 29073 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:57.497830 29073 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:57.498873 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:57.493997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35005 (local address 127.0.0.1:39797)
0928 12:34:57.494035 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:57.494037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:57.498002 (+ 3965us) server_negotiation.cc:407] Connection header received
0928 12:34:57.498069 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:57.498069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:57.498075 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:57.498120 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:57.498673 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:57.498674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:57.498674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:57.498678 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:57.498693 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:57.498698 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:57.498745 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:57.498792 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 12:34:57.498807 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:34:57.501348 29073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04d56cb4b8c4440d94fe0ce5a306f51f: Generated new TSK 0
I20260928 12:34:57.501515 29073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:34:57.504197 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 28995
2026-09-28T12:34:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1184 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:57 dist-test-slave-g31s krb5kdc[29094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:57 dist-test-slave-g31s krb5kdc[29094](info): set up 2 sockets
Sep 28 12:34:57 dist-test-slave-g31s krb5kdc[29094](info): commencing operation
krb5kdc: starting...
W20260928 12:34:58.445683 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:34:58 dist-test-slave-g31s krb5kdc[29094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:34:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:34:58.557251 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:58.639874 29107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:58.640127 29107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:58.640163 29107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:58.641454 29107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:58.641510 29107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:58.641528 29107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:58.641546 29107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:58.641561 29107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:58.643304 29107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:58.643582 29107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:58.643805 29107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:34:58.648715 29107 server_base.cc:1034] running on GCE node
W20260928 12:34:58.648698 29112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:58.648677 29115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:58.648677 29113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:58.649183 29107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:58.649482 29107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:58.650628 29107 hybrid_clock.cc:648] HybridClock initialized: now 1790598898650609 us; error 33 us; skew 500 ppm
Sep 28 12:34:58 dist-test-slave-g31s krb5kdc[29094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:58.658301 29107 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:34:58.658782 29107 webserver.cc:460] Webserver started at http://127.0.0.1:34581/ using document root <none> and password file <none>
I20260928 12:34:58.659014 29107 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:58.659060 29107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:58.659188 29107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:58.660292 29107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "ae6a4a9f7f27491a9820d997e10a432e"
format_stamp: "Formatted at 2026-09-28 12:34:58 on dist-test-slave-g31s"
server_key: "652d408815e48fa0c5cfb646f0db873c"
server_key_iv: "00a403296007e4b2228794fcf8183d7d"
server_key_version: "encryptionkey@0"
I20260928 12:34:58.660630 29107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "ae6a4a9f7f27491a9820d997e10a432e"
format_stamp: "Formatted at 2026-09-28 12:34:58 on dist-test-slave-g31s"
server_key: "652d408815e48fa0c5cfb646f0db873c"
server_key_iv: "00a403296007e4b2228794fcf8183d7d"
server_key_version: "encryptionkey@0"
I20260928 12:34:58.662065 29107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:34:58.662789 29122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:58.662935 29107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:34:58.663040 29107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "ae6a4a9f7f27491a9820d997e10a432e"
format_stamp: "Formatted at 2026-09-28 12:34:58 on dist-test-slave-g31s"
server_key: "652d408815e48fa0c5cfb646f0db873c"
server_key_iv: "00a403296007e4b2228794fcf8183d7d"
server_key_version: "encryptionkey@0"
I20260928 12:34:58.663108 29107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:58.672565 29107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:58.674383 29107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:58.674525 29107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:58.678346 29107 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41197
I20260928 12:34:58.678383 29174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41197 every 8 connection(s)
I20260928 12:34:58.678712 29107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:58.679646 29175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:58.681953 29175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e: Bootstrap starting.
I20260928 12:34:58.682021 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 29107
I20260928 12:34:58.682139 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:58.682341 8244 external_mini_cluster.cc:1428] Setting key 4f076aa23fcea58aefe59c6cdaf1ad16
I20260928 12:34:58.682684 29175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:58.682991 29175 log.cc:824] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:58.683663 29175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e: No bootstrap required, opened a new log
I20260928 12:34:58.685443 29175 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6a4a9f7f27491a9820d997e10a432e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260928 12:34:58.685585 29175 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:58.685607 29175 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6a4a9f7f27491a9820d997e10a432e, State: Initialized, Role: FOLLOWER
I20260928 12:34:58.685731 29175 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6a4a9f7f27491a9820d997e10a432e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260928 12:34:58.685791 29175 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:58.685849 29175 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:58.685894 29175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:58.686447 29175 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6a4a9f7f27491a9820d997e10a432e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260928 12:34:58.686548 29175 leader_election.cc:302] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6a4a9f7f27491a9820d997e10a432e; no voters:
I20260928 12:34:58.686714 29175 leader_election.cc:288] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:58.686751 29180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:58.686928 29180 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [term 1 LEADER]: Becoming Leader. State: Replica: ae6a4a9f7f27491a9820d997e10a432e, State: Running, Role: LEADER
I20260928 12:34:58.687011 29180 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6a4a9f7f27491a9820d997e10a432e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260928 12:34:58.687319 29180 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae6a4a9f7f27491a9820d997e10a432e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6a4a9f7f27491a9820d997e10a432e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } } }
I20260928 12:34:58.687458 29180 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:58.687477 29182 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6a4a9f7f27491a9820d997e10a432e. Latest consensus state: current_term: 1 leader_uuid: "ae6a4a9f7f27491a9820d997e10a432e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6a4a9f7f27491a9820d997e10a432e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } } }
I20260928 12:34:58.687604 29182 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:58.687393 29175 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 12:34:58 dist-test-slave-g31s krb5kdc[29094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:34:58.688938 29184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:58.689541 29184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:58.691850 29184 catalog_manager.cc:1347] Generated new cluster ID: 999764a91973412f9df66849a62d4185
I20260928 12:34:58.691958 29184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:58.693008 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:58.682853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:47597 (local address 127.0.0.1:41197)
0928 12:34:58.683046 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:58.683056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:58.683094 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:34:58.683632 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:58.683640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:58.683693 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:58.683832 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:58.684422 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:58.685338 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:58.686250 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:58.686706 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:58.686810 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:58.689325 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:58.689330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:58.689331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:58.689347 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:58.691275 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:58.691732 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:58.691735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:58.691735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:58.691842 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:58.692115 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:58.692116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:58.692116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:58.692410 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:58.692570 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:58.692710 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 12:34:58.692808 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260928 12:34:58.693961 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:34:58.702654 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:58.695653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:45409 (local address 127.0.0.1:41197)
0928 12:34:58.695689 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:58.695691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:58.695699 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:34:58.695719 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:58.695719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:58.695729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:58.695779 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:58.696403 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:58.697148 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:58.698171 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:58.698500 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:58.698555 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:58.699996 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:58.699997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:58.699998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:34:58.700006 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:58.701496 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:58.701931 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:58.701932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:58.701932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:58.701985 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:34:58.702177 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:34:58.702178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:34:58.702178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:34:58.702292 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:58.702375 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:58.702502 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 12:34:58.702559 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:34:58.706521 29184 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:58.707029 29184 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:58.709507 29184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae6a4a9f7f27491a9820d997e10a432e: Generated new TSK 0
I20260928 12:34:58.709676 29184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:34:58.717186 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 29107
W20260928 12:34:58.720906 29092 connection.cc:535] client connection to 127.0.0.1:41197 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41197): Connection reset by peer (error 104)
2026-09-28T12:34:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:58 dist-test-slave-g31s krb5kdc[29206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:58 dist-test-slave-g31s krb5kdc[29206](info): set up 2 sockets
Sep 28 12:34:58 dist-test-slave-g31s krb5kdc[29206](info): commencing operation
krb5kdc: starting...
W20260928 12:34:59.661654 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:34:59 dist-test-slave-g31s krb5kdc[29206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:34:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:34:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:34:59.777526 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:34:59.857478 29219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:34:59.857751 29219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:34:59.857787 29219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:34:59.859193 29219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:34:59.859263 29219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:34:59.859282 29219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:34:59.859299 29219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:34:59.859314 29219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:34:59.861084 29219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:34:59.861394 29219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:34:59.861624 29219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:34:59.866562 29225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:59.866562 29227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:34:59.866585 29224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:34:59.866845 29219 server_base.cc:1034] running on GCE node
I20260928 12:34:59.867106 29219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:34:59.867391 29219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:34:59.868552 29219 hybrid_clock.cc:648] HybridClock initialized: now 1790598899868536 us; error 27 us; skew 500 ppm
Sep 28 12:34:59 dist-test-slave-g31s krb5kdc[29206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:34:59.871441 29219 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:34:59.871912 29219 webserver.cc:460] Webserver started at http://127.0.0.1:33585/ using document root <none> and password file <none>
I20260928 12:34:59.872146 29219 fs_manager.cc:359] Metadata directory not provided
I20260928 12:34:59.872195 29219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:34:59.872294 29219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:34:59.873370 29219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "b572df3a2fd146abafb9b5b9fc286a68"
format_stamp: "Formatted at 2026-09-28 12:34:59 on dist-test-slave-g31s"
server_key: "400ef694f59050ed1ff9a690046b78ac"
server_key_iv: "cb144a21d68831142fe2da61ef0bf02b"
server_key_version: "encryptionkey@0"
I20260928 12:34:59.873687 29219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "b572df3a2fd146abafb9b5b9fc286a68"
format_stamp: "Formatted at 2026-09-28 12:34:59 on dist-test-slave-g31s"
server_key: "400ef694f59050ed1ff9a690046b78ac"
server_key_iv: "cb144a21d68831142fe2da61ef0bf02b"
server_key_version: "encryptionkey@0"
I20260928 12:34:59.875105 29219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:34:59.875842 29234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:34:59.876009 29219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:34:59.876093 29219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "b572df3a2fd146abafb9b5b9fc286a68"
format_stamp: "Formatted at 2026-09-28 12:34:59 on dist-test-slave-g31s"
server_key: "400ef694f59050ed1ff9a690046b78ac"
server_key_iv: "cb144a21d68831142fe2da61ef0bf02b"
server_key_version: "encryptionkey@0"
I20260928 12:34:59.876157 29219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:34:59.900316 29219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:34:59.902494 29219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:34:59.902618 29219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:34:59.906348 29219 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44855
I20260928 12:34:59.906390 29286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44855 every 8 connection(s)
I20260928 12:34:59.906720 29219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:34:59.907703 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:34:59.909937 29287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68: Bootstrap starting.
I20260928 12:34:59.910465 29287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68: Neither blocks nor log segments found. Creating new log.
I20260928 12:34:59.910742 29287 log.cc:824] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68: Log is configured to *not* fsync() on all Append() calls
I20260928 12:34:59.911377 29287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68: No bootstrap required, opened a new log
I20260928 12:34:59.912482 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 29219
I20260928 12:34:59.912612 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:34:59.912794 8244 external_mini_cluster.cc:1428] Setting key 6a24dcbedfba7ac735d38cba2e415286
I20260928 12:34:59.913211 29287 raft_consensus.cc:348] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b572df3a2fd146abafb9b5b9fc286a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260928 12:34:59.913394 29287 raft_consensus.cc:374] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:34:59.913475 29287 raft_consensus.cc:729] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b572df3a2fd146abafb9b5b9fc286a68, State: Initialized, Role: FOLLOWER
I20260928 12:34:59.913612 29287 consensus_queue.cc:260] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b572df3a2fd146abafb9b5b9fc286a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260928 12:34:59.913703 29287 raft_consensus.cc:388] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:34:59.913741 29287 raft_consensus.cc:482] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:34:59.913787 29287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:34:59.914929 29287 raft_consensus.cc:504] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b572df3a2fd146abafb9b5b9fc286a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260928 12:34:59.915040 29287 leader_election.cc:302] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b572df3a2fd146abafb9b5b9fc286a68; no voters:
I20260928 12:34:59.915180 29287 leader_election.cc:288] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:34:59.915230 29292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:34:59.915401 29292 raft_consensus.cc:686] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [term 1 LEADER]: Becoming Leader. State: Replica: b572df3a2fd146abafb9b5b9fc286a68, State: Running, Role: LEADER
I20260928 12:34:59.915405 29287 sys_catalog.cc:564] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:34:59.915526 29292 consensus_queue.cc:237] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b572df3a2fd146abafb9b5b9fc286a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260928 12:34:59.915876 29293 sys_catalog.cc:455] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b572df3a2fd146abafb9b5b9fc286a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b572df3a2fd146abafb9b5b9fc286a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } } }
I20260928 12:34:59.915894 29294 sys_catalog.cc:455] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b572df3a2fd146abafb9b5b9fc286a68. Latest consensus state: current_term: 1 leader_uuid: "b572df3a2fd146abafb9b5b9fc286a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b572df3a2fd146abafb9b5b9fc286a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } } }
I20260928 12:34:59.915958 29293 sys_catalog.cc:458] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:59.915966 29294 sys_catalog.cc:458] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68 [sys.catalog]: This master's current role is: LEADER
I20260928 12:34:59.916153 29296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:34:59.916651 29296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:34:59.918223 29296 catalog_manager.cc:1347] Generated new cluster ID: 1870c1173b8f44b7b2c292f0b58419f9
I20260928 12:34:59.918269 29296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:34:59.918812 29291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:59.913341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:45803 (local address 127.0.0.1:44855)
0928 12:34:59.913535 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:59.913544 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:59.913577 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:34:59.913693 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:59.913699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:59.913761 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:59.913932 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:59.914134 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:59.915495 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:59.916621 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:59.917264 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:59.917479 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:59.918088 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:59.918091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:59.918091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:59.918312 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:59.918363 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:59.918379 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:59.918466 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:59.918572 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 12:34:59.918598 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260928 12:34:59.922855 29296 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:34:59.922837 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:34:59.923475 29296 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:34:59.927796 29296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b572df3a2fd146abafb9b5b9fc286a68: Generated new TSK 0
I20260928 12:34:59.927958 29291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:34:59.924285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:57009 (local address 127.0.0.1:44855)
0928 12:34:59.924323 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:34:59.924324 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:34:59.924442 (+ 118us) server_negotiation.cc:407] Connection header received
0928 12:34:59.924491 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:34:59.924492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:34:59.924504 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:34:59.924569 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:34:59.924700 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:59.925443 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:59.926450 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:34:59.926782 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:34:59.926853 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:34:59.927698 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:34:59.927699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:34:59.927699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:34:59.927705 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:34:59.927716 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:34:59.927720 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:34:59.927767 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:34:59.927867 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 12:34:59.927880 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:34:59.928086 29296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:34:59.932205 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 29219
W20260928 12:34:59.935997 29204 connection.cc:535] client connection to 127.0.0.1:44855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44855): Connection reset by peer (error 104)
2026-09-28T12:34:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:34:59 dist-test-slave-g31s krb5kdc[29318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:34:59 dist-test-slave-g31s krb5kdc[29318](info): set up 2 sockets
Sep 28 12:34:59 dist-test-slave-g31s krb5kdc[29318](info): commencing operation
krb5kdc: starting...
W20260928 12:35:00.881836 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:00 dist-test-slave-g31s krb5kdc[29318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:35:00.996803 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:01.078050 29331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:01.078275 29331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:01.078310 29331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:01.079528 29331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:01.079579 29331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:01.079597 29331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:01.079617 29331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:01.079632 29331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:01.081220 29331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:01.081480 29331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:01.081699 29331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:01.086539 29336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:01.086539 29337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:01.086525 29339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:01.086779 29331 server_base.cc:1034] running on GCE node
I20260928 12:35:01.087004 29331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:01.087307 29331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:01.088478 29331 hybrid_clock.cc:648] HybridClock initialized: now 1790598901088463 us; error 30 us; skew 500 ppm
Sep 28 12:35:01 dist-test-slave-g31s krb5kdc[29318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:01.091131 29331 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:35:01.091601 29331 webserver.cc:460] Webserver started at http://127.0.0.1:45785/ using document root <none> and password file <none>
I20260928 12:35:01.091835 29331 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:01.091885 29331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:01.092011 29331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:01.093117 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "69e89ba0402a45ceb12669de7fedbb5c"
format_stamp: "Formatted at 2026-09-28 12:35:01 on dist-test-slave-g31s"
server_key: "bf00bf41f34912eab8c63f2ded91412f"
server_key_iv: "19010e7fc39520cfbbb7e72fb6d2c703"
server_key_version: "encryptionkey@0"
I20260928 12:35:01.093462 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "69e89ba0402a45ceb12669de7fedbb5c"
format_stamp: "Formatted at 2026-09-28 12:35:01 on dist-test-slave-g31s"
server_key: "bf00bf41f34912eab8c63f2ded91412f"
server_key_iv: "19010e7fc39520cfbbb7e72fb6d2c703"
server_key_version: "encryptionkey@0"
I20260928 12:35:01.094919 29331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:35:01.095734 29346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:01.095906 29331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:35:01.095995 29331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "69e89ba0402a45ceb12669de7fedbb5c"
format_stamp: "Formatted at 2026-09-28 12:35:01 on dist-test-slave-g31s"
server_key: "bf00bf41f34912eab8c63f2ded91412f"
server_key_iv: "19010e7fc39520cfbbb7e72fb6d2c703"
server_key_version: "encryptionkey@0"
I20260928 12:35:01.096060 29331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:01.099529 29331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:01.101573 29331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:01.101709 29331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:01.105552 29331 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38737
I20260928 12:35:01.105587 29398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38737 every 8 connection(s)
I20260928 12:35:01.105962 29331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:01.106891 29399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:01.109161 29399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c: Bootstrap starting.
I20260928 12:35:01.109694 29399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:01.109969 29399 log.cc:824] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:01.110628 29399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c: No bootstrap required, opened a new log
I20260928 12:35:01.111387 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 29331
I20260928 12:35:01.111510 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:01.111665 8244 external_mini_cluster.cc:1428] Setting key 952a956bd96338c092ec1507c7bb6b05
I20260928 12:35:01.112505 29399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e89ba0402a45ceb12669de7fedbb5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } }
I20260928 12:35:01.112673 29399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:35:01.112711 29399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e89ba0402a45ceb12669de7fedbb5c, State: Initialized, Role: FOLLOWER
I20260928 12:35:01.112859 29399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e89ba0402a45ceb12669de7fedbb5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } }
I20260928 12:35:01.113215 29399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:35:01.113263 29399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:35:01.113299 29399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:01.113762 29403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:01.112169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:37443 (local address 127.0.0.1:38737)
0928 12:35:01.112371 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:01.112382 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:01.112417 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:35:01.112543 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:01.112550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:01.112580 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:01.112726 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:01.113370 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:01.113374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:01.113374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:01.113379 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:01.113422 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:01.113433 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:01.113486 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:01.113560 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 12:35:01.113603 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260928 12:35:01.114329 29399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e89ba0402a45ceb12669de7fedbb5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } }
I20260928 12:35:01.114434 29399 leader_election.cc:302] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e89ba0402a45ceb12669de7fedbb5c; no voters:
I20260928 12:35:01.114607 29399 leader_election.cc:288] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:35:01.114642 29404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:01.114673 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:35:01.114773 29404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [term 1 LEADER]: Becoming Leader. State: Replica: 69e89ba0402a45ceb12669de7fedbb5c, State: Running, Role: LEADER
I20260928 12:35:01.114847 29399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:01.114904 29404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e89ba0402a45ceb12669de7fedbb5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } }
I20260928 12:35:01.115365 29405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69e89ba0402a45ceb12669de7fedbb5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e89ba0402a45ceb12669de7fedbb5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } } }
I20260928 12:35:01.115381 29406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e89ba0402a45ceb12669de7fedbb5c. Latest consensus state: current_term: 1 leader_uuid: "69e89ba0402a45ceb12669de7fedbb5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e89ba0402a45ceb12669de7fedbb5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } } }
I20260928 12:35:01.115459 29405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:01.115468 29406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:01.115665 29413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:01.116206 29413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:01.117532 29403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:01.116408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:47295 (local address 127.0.0.1:38737)
0928 12:35:01.116462 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:01.116464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:01.116601 (+ 137us) server_negotiation.cc:407] Connection header received
0928 12:35:01.116673 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:01.116674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:01.116679 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:01.116729 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:01.117321 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:01.117323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:01.117323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:01.117327 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:01.117343 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:01.117347 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:01.117401 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:01.117446 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 12:35:01.117462 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:35:01.122746 29413 catalog_manager.cc:1347] Generated new cluster ID: 83145586c41b4029b5ea07a86e92b5f1
I20260928 12:35:01.122807 29413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:35:01.130964 29413 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:01.131129 29413 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:01.136858 29413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69e89ba0402a45ceb12669de7fedbb5c: Generated new TSK 0
I20260928 12:35:01.137087 29413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:35:01.142714 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 29331
2026-09-28T12:35:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:35:01 dist-test-slave-g31s krb5kdc[29430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:35:01 dist-test-slave-g31s krb5kdc[29430](info): set up 2 sockets
Sep 28 12:35:01 dist-test-slave-g31s krb5kdc[29430](info): commencing operation
krb5kdc: starting...
W20260928 12:35:02.101670 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:02 dist-test-slave-g31s krb5kdc[29430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:35:02.218739 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:02.299844 29443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:02.300081 29443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:02.300114 29443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:02.301353 29443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:02.301402 29443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:02.301421 29443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:02.301436 29443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:02.301452 29443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:02.303112 29443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:02.303371 29443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:02.303582 29443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:02.308429 29449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:02.308445 29448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:02.308450 29451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:02.308665 29443 server_base.cc:1034] running on GCE node
I20260928 12:35:02.308990 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:02.309371 29443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:02.310580 29443 hybrid_clock.cc:648] HybridClock initialized: now 1790598902310556 us; error 38 us; skew 500 ppm
Sep 28 12:35:02 dist-test-slave-g31s krb5kdc[29430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:02.313652 29443 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:35:02.314252 29443 webserver.cc:460] Webserver started at http://127.0.0.1:41397/ using document root <none> and password file <none>
I20260928 12:35:02.314523 29443 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:02.314582 29443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:02.314713 29443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:02.316159 29443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "44078ae1c8f240bfad23fd06e404e9db"
format_stamp: "Formatted at 2026-09-28 12:35:02 on dist-test-slave-g31s"
server_key: "497efd94cacd3169606969c659213e42"
server_key_iv: "a0b7b98f173c4d919b45d1b605706016"
server_key_version: "encryptionkey@0"
I20260928 12:35:02.316574 29443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "44078ae1c8f240bfad23fd06e404e9db"
format_stamp: "Formatted at 2026-09-28 12:35:02 on dist-test-slave-g31s"
server_key: "497efd94cacd3169606969c659213e42"
server_key_iv: "a0b7b98f173c4d919b45d1b605706016"
server_key_version: "encryptionkey@0"
I20260928 12:35:02.318262 29443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 12:35:02.319113 29458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:02.319276 29443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:35:02.319370 29443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "44078ae1c8f240bfad23fd06e404e9db"
format_stamp: "Formatted at 2026-09-28 12:35:02 on dist-test-slave-g31s"
server_key: "497efd94cacd3169606969c659213e42"
server_key_iv: "a0b7b98f173c4d919b45d1b605706016"
server_key_version: "encryptionkey@0"
I20260928 12:35:02.319442 29443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:02.347434 29443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:02.350240 29443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:02.350412 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:02.355697 29443 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45045
I20260928 12:35:02.355734 29510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45045 every 8 connection(s)
I20260928 12:35:02.356172 29443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:02.357227 29511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:02.359678 29511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db: Bootstrap starting.
I20260928 12:35:02.360276 29511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:02.360560 29511 log.cc:824] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:02.361382 29511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db: No bootstrap required, opened a new log
I20260928 12:35:02.363181 29511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44078ae1c8f240bfad23fd06e404e9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 12:35:02.363354 29511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:35:02.363386 29511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44078ae1c8f240bfad23fd06e404e9db, State: Initialized, Role: FOLLOWER
I20260928 12:35:02.363540 29511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44078ae1c8f240bfad23fd06e404e9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 12:35:02.363626 29511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:35:02.363660 29511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:35:02.363710 29511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:02.364367 29511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44078ae1c8f240bfad23fd06e404e9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 12:35:02.364436 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 29443
I20260928 12:35:02.364476 29511 leader_election.cc:302] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44078ae1c8f240bfad23fd06e404e9db; no voters:
I20260928 12:35:02.364574 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:02.364666 29511 leader_election.cc:288] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:35:02.364718 29514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:02.364741 8244 external_mini_cluster.cc:1428] Setting key 6354d7bee0e71b434a4343ec730b1468
I20260928 12:35:02.364836 29514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [term 1 LEADER]: Becoming Leader. State: Replica: 44078ae1c8f240bfad23fd06e404e9db, State: Running, Role: LEADER
I20260928 12:35:02.364915 29514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44078ae1c8f240bfad23fd06e404e9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 12:35:02.364969 29511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:02.365391 29514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44078ae1c8f240bfad23fd06e404e9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44078ae1c8f240bfad23fd06e404e9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } } }
I20260928 12:35:02.365494 29514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:02.365612 29514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44078ae1c8f240bfad23fd06e404e9db. Latest consensus state: current_term: 1 leader_uuid: "44078ae1c8f240bfad23fd06e404e9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44078ae1c8f240bfad23fd06e404e9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } } }
I20260928 12:35:02.365743 29514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:02.365911 29520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:02.366549 29520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:02.368167 29520 catalog_manager.cc:1347] Generated new cluster ID: c8688f5855ac4d1b866938736a6a884e
I20260928 12:35:02.368219 29520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:35:02 dist-test-slave-g31s krb5kdc[29430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:35:02.377712 29519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:02.365247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:52751 (local address 127.0.0.1:45045)
0928 12:35:02.365739 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:02.365752 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:02.365792 (+ 40us) server_negotiation.cc:407] Connection header received
0928 12:35:02.365957 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:02.365965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:02.366042 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:02.367041 (+ 999us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:02.367147 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:02.368526 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:02.369620 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:02.370301 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:02.370423 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:02.373331 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:02.373337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:02.373337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:02.373353 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:02.375938 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:02.376439 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:02.376443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:02.376443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:02.376552 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:02.376781 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:02.376782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:02.376782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:02.377093 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:02.377229 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:02.377363 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 12:35:02.377464 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":79,"threads_started":1}
I20260928 12:35:02.378785 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:35:02.387668 29519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:02.380347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:56331 (local address 127.0.0.1:45045)
0928 12:35:02.380398 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:02.380400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:02.380504 (+ 104us) server_negotiation.cc:407] Connection header received
0928 12:35:02.380554 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:02.380555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:02.380566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:02.380633 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:02.381208 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:02.382030 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:02.383003 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:02.383341 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:02.383413 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:02.384852 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:02.384854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:02.384854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:02.384863 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:02.386453 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:02.386853 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:02.386854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:02.386854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:02.386918 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:02.387151 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:02.387152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:02.387152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:02.387292 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:02.387395 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:02.387496 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 12:35:02.387563 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:35:02.400936 29520 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:02.401487 29520 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:02.405424 29520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44078ae1c8f240bfad23fd06e404e9db: Generated new TSK 0
I20260928 12:35:02.405632 29520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:35:02.408517 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 29443
W20260928 12:35:02.412648 29428 connection.cc:535] client connection to 127.0.0.1:45045 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45045): Connection reset by peer (error 104)
2026-09-28T12:35:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:35:02 dist-test-slave-g31s krb5kdc[29542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:35:02 dist-test-slave-g31s krb5kdc[29542](info): set up 2 sockets
Sep 28 12:35:02 dist-test-slave-g31s krb5kdc[29542](info): commencing operation
krb5kdc: starting...
W20260928 12:35:03.361896 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:03 dist-test-slave-g31s krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:35:03.478775 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:03.560231 29555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:03.560479 29555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:03.560513 29555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:03.561786 29555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:03.561879 29555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:03.561897 29555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:03.561913 29555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:03.561929 29555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:03.564097 29555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:03.564469 29555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:03.564710 29555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:03.569571 29563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:03.569592 29561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:03.569756 29555 server_base.cc:1034] running on GCE node
W20260928 12:35:03.569576 29560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:03.570089 29555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:03.570429 29555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:03.571570 29555 hybrid_clock.cc:648] HybridClock initialized: now 1790598903571555 us; error 29 us; skew 500 ppm
Sep 28 12:35:03 dist-test-slave-g31s krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:03.574086 29555 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:35:03.574579 29555 webserver.cc:460] Webserver started at http://127.0.0.1:45397/ using document root <none> and password file <none>
I20260928 12:35:03.574800 29555 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:03.574849 29555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:03.574978 29555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:03.576042 29555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "2e62b07f6e4c440986e1133613ce452f"
format_stamp: "Formatted at 2026-09-28 12:35:03 on dist-test-slave-g31s"
server_key: "c914ae0f0d7c89c02cbbc97bc6c1090a"
server_key_iv: "41e0aa8c79157020364d1beea6eed8ef"
server_key_version: "encryptionkey@0"
I20260928 12:35:03.576386 29555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "2e62b07f6e4c440986e1133613ce452f"
format_stamp: "Formatted at 2026-09-28 12:35:03 on dist-test-slave-g31s"
server_key: "c914ae0f0d7c89c02cbbc97bc6c1090a"
server_key_iv: "41e0aa8c79157020364d1beea6eed8ef"
server_key_version: "encryptionkey@0"
I20260928 12:35:03.577677 29555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:35:03.578500 29570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:03.578652 29555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:35:03.578732 29555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "2e62b07f6e4c440986e1133613ce452f"
format_stamp: "Formatted at 2026-09-28 12:35:03 on dist-test-slave-g31s"
server_key: "c914ae0f0d7c89c02cbbc97bc6c1090a"
server_key_iv: "41e0aa8c79157020364d1beea6eed8ef"
server_key_version: "encryptionkey@0"
I20260928 12:35:03.578802 29555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:03.605769 29555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:03.607939 29555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:03.608080 29555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:03.611756 29555 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36405
I20260928 12:35:03.611796 29622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36405 every 8 connection(s)
I20260928 12:35:03.612110 29555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:03.612920 29623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:03.613714 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 29555
I20260928 12:35:03.613842 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:03.614010 8244 external_mini_cluster.cc:1428] Setting key e33e84252756a3ea0691e351eceb2320
I20260928 12:35:03.616005 29623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f: Bootstrap starting.
I20260928 12:35:03.616705 29623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:03.616968 29623 log.cc:824] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:03.617789 29623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f: No bootstrap required, opened a new log
I20260928 12:35:03.620249 29623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e62b07f6e4c440986e1133613ce452f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260928 12:35:03.620427 29623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:35:03.620446 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:03.614477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:55981 (local address 127.0.0.1:36405)
0928 12:35:03.614643 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:03.614652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:03.614683 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:35:03.615152 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:03.615161 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:03.615247 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:03.615386 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:03.615730 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:03.617022 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:03.618023 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:03.618672 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:03.618780 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:03.619686 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:03.619690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:03.619690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:03.619894 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:03.619932 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:03.619946 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:03.620035 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:03.620167 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 12:35:03.620197 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260928 12:35:03.620502 29623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e62b07f6e4c440986e1133613ce452f, State: Initialized, Role: FOLLOWER
I20260928 12:35:03.620718 29623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e62b07f6e4c440986e1133613ce452f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260928 12:35:03.620798 29623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:35:03.620850 29623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:35:03.620898 29623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:03.621587 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:35:03.621587 29623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e62b07f6e4c440986e1133613ce452f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260928 12:35:03.621686 29623 leader_election.cc:302] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e62b07f6e4c440986e1133613ce452f; no voters:
I20260928 12:35:03.621853 29623 leader_election.cc:288] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:35:03.621891 29628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:03.622021 29628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [term 1 LEADER]: Becoming Leader. State: Replica: 2e62b07f6e4c440986e1133613ce452f, State: Running, Role: LEADER
I20260928 12:35:03.622097 29623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:03.622108 29628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e62b07f6e4c440986e1133613ce452f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260928 12:35:03.622491 29630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e62b07f6e4c440986e1133613ce452f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e62b07f6e4c440986e1133613ce452f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } } }
I20260928 12:35:03.622630 29630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:03.622475 29628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e62b07f6e4c440986e1133613ce452f. Latest consensus state: current_term: 1 leader_uuid: "2e62b07f6e4c440986e1133613ce452f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e62b07f6e4c440986e1133613ce452f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } } }
I20260928 12:35:03.622839 29628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f [sys.catalog]: This master's current role is: LEADER
W20260928 12:35:03.624202 29649 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:35:03.624296 29649 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:35:03.624477 29640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:03.624650 29640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:03.627235 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:03.623572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:35115 (local address 127.0.0.1:36405)
0928 12:35:03.623605 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:03.623607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:03.623993 (+ 386us) server_negotiation.cc:407] Connection header received
0928 12:35:03.624021 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:03.624022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:03.624033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:03.624087 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:03.624267 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:03.624951 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:03.625934 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:03.626233 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:03.626274 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:03.626978 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:03.626980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:03.626980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:03.626984 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:03.627000 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:03.627004 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:03.627068 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:03.627158 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 12:35:03.627169 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:35:03.627365 29640 catalog_manager.cc:1347] Generated new cluster ID: 77993447b0734f769a05d4b36bc10513
I20260928 12:35:03.627408 29640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:35:03.644315 29640 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:03.644941 29640 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:03.651136 29640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e62b07f6e4c440986e1133613ce452f: Generated new TSK 0
I20260928 12:35:03.651336 29640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:35:03.654851 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 29555
W20260928 12:35:03.658958 29540 connection.cc:535] client connection to 127.0.0.1:36405 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36405): Connection reset by peer (error 104)
2026-09-28T12:35:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1246 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:35:03 dist-test-slave-g31s krb5kdc[29654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:35:03 dist-test-slave-g31s krb5kdc[29654](info): set up 2 sockets
Sep 28 12:35:03 dist-test-slave-g31s krb5kdc[29654](info): commencing operation
krb5kdc: starting...
W20260928 12:35:04.613673 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:04 dist-test-slave-g31s krb5kdc[29654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:35:04.724915 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:04.805325 29667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:04.805575 29667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:04.805612 29667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:04.806905 29667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:04.806957 29667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:04.806977 29667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:04.806993 29667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:04.807008 29667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:04.808604 29667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:04.808883 29667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:04.809121 29667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:04.813872 29672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:04.813935 29667 server_base.cc:1034] running on GCE node
W20260928 12:35:04.813920 29675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:04.813912 29673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:04.814358 29667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:04.814663 29667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:04.815832 29667 hybrid_clock.cc:648] HybridClock initialized: now 1790598904815808 us; error 37 us; skew 500 ppm
Sep 28 12:35:04 dist-test-slave-g31s krb5kdc[29654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:04.818250 29667 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:35:04.818706 29667 webserver.cc:460] Webserver started at http://127.0.0.1:33495/ using document root <none> and password file <none>
I20260928 12:35:04.818915 29667 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:04.818965 29667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:04.819090 29667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:04.820119 29667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "6fa4ce10982a48c6a9df7a6e4ba9a38b"
format_stamp: "Formatted at 2026-09-28 12:35:04 on dist-test-slave-g31s"
server_key: "ec0938249428c37935b015940318ea0e"
server_key_iv: "6049e82b71769d02de0d0610d093058a"
server_key_version: "encryptionkey@0"
I20260928 12:35:04.820449 29667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "6fa4ce10982a48c6a9df7a6e4ba9a38b"
format_stamp: "Formatted at 2026-09-28 12:35:04 on dist-test-slave-g31s"
server_key: "ec0938249428c37935b015940318ea0e"
server_key_iv: "6049e82b71769d02de0d0610d093058a"
server_key_version: "encryptionkey@0"
I20260928 12:35:04.821682 29667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:35:04.822431 29682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:04.822582 29667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:04.822660 29667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "6fa4ce10982a48c6a9df7a6e4ba9a38b"
format_stamp: "Formatted at 2026-09-28 12:35:04 on dist-test-slave-g31s"
server_key: "ec0938249428c37935b015940318ea0e"
server_key_iv: "6049e82b71769d02de0d0610d093058a"
server_key_version: "encryptionkey@0"
I20260928 12:35:04.822727 29667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:04.825914 29667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:04.827677 29667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:04.827800 29667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:04.831583 29667 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37815
I20260928 12:35:04.831624 29734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37815 every 8 connection(s)
I20260928 12:35:04.831928 29667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:04.832851 29735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:04.835263 29735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b: Bootstrap starting.
I20260928 12:35:04.835822 29735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:04.836094 29735 log.cc:824] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:04.836817 29735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b: No bootstrap required, opened a new log
I20260928 12:35:04.838541 29735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa4ce10982a48c6a9df7a6e4ba9a38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37815 } }
I20260928 12:35:04.838693 29735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:35:04.838716 29735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa4ce10982a48c6a9df7a6e4ba9a38b, State: Initialized, Role: FOLLOWER
I20260928 12:35:04.838861 29735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa4ce10982a48c6a9df7a6e4ba9a38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37815 } }
I20260928 12:35:04.838935 29735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:35:04.838969 29735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:35:04.839015 29735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:04.839249 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 29667
I20260928 12:35:04.839353 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:04.839514 8244 external_mini_cluster.cc:1428] Setting key c623120ebe02e9531f9a3fbe2932c024
I20260928 12:35:04.839664 29735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa4ce10982a48c6a9df7a6e4ba9a38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37815 } }
I20260928 12:35:04.839769 29735 leader_election.cc:302] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fa4ce10982a48c6a9df7a6e4ba9a38b; no voters:
I20260928 12:35:04.839953 29735 leader_election.cc:288] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:35:04.839980 29738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:04.840158 29738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [term 1 LEADER]: Becoming Leader. State: Replica: 6fa4ce10982a48c6a9df7a6e4ba9a38b, State: Running, Role: LEADER
I20260928 12:35:04.840247 29738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa4ce10982a48c6a9df7a6e4ba9a38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37815 } }
I20260928 12:35:04.840297 29735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:04.840561 29738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fa4ce10982a48c6a9df7a6e4ba9a38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa4ce10982a48c6a9df7a6e4ba9a38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37815 } } }
I20260928 12:35:04.840649 29738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:04.840765 29738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fa4ce10982a48c6a9df7a6e4ba9a38b. Latest consensus state: current_term: 1 leader_uuid: "6fa4ce10982a48c6a9df7a6e4ba9a38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa4ce10982a48c6a9df7a6e4ba9a38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37815 } } }
I20260928 12:35:04.840848 29744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:04.840871 29738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:04.841486 29744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:04.843132 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:04.840017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:45813 (local address 127.0.0.1:37815)
0928 12:35:04.841216 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:04.841228 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:04.841272 (+ 44us) server_negotiation.cc:407] Connection header received
0928 12:35:04.841401 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:04.841409 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:04.841442 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:04.842115 (+ 673us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:04.842595 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:04.842598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:04.842598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:04.842601 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:04.842636 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:04.842652 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:04.842718 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:04.842886 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 12:35:04.842934 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1137,"thread_start_us":86,"threads_started":1}
I20260928 12:35:04.844044 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:35:04.846240 29744 catalog_manager.cc:1347] Generated new cluster ID: ff765c0409de49d4969c921b2f990d73
I20260928 12:35:04.846295 29744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:35:04.846448 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:04.845352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:48867 (local address 127.0.0.1:37815)
0928 12:35:04.845397 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:04.845399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:04.845525 (+ 126us) server_negotiation.cc:407] Connection header received
0928 12:35:04.845590 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:04.845591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:04.845607 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:04.845668 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:04.846244 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:04.846246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:04.846246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:04.846250 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:04.846265 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:04.846269 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:04.846316 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:04.846361 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 12:35:04.846374 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:35:04.879047 29744 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:04.879274 29744 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:04.888563 29744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fa4ce10982a48c6a9df7a6e4ba9a38b: Generated new TSK 0
I20260928 12:35:04.888790 29744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:35:04.892481 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 29667
2026-09-28T12:35:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1237 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:35:04 dist-test-slave-g31s krb5kdc[29766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:35:04 dist-test-slave-g31s krb5kdc[29766](info): set up 2 sockets
Sep 28 12:35:04 dist-test-slave-g31s krb5kdc[29766](info): commencing operation
krb5kdc: starting...
W20260928 12:35:05.829614 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:05 dist-test-slave-g31s krb5kdc[29766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:35:05.944334 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:35261
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:35261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:06.025244 29779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:06.025440 29779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:06.025461 29779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:06.026612 29779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:06.026656 29779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:06.026669 29779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:06.026680 29779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:06.026690 29779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:06.028121 29779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:35261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:06.028332 29779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:06.028554 29779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:06.032956 29787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:06.033015 29785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:06.032953 29784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:06.033308 29779 server_base.cc:1034] running on GCE node
I20260928 12:35:06.033527 29779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:06.033772 29779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:06.034950 29779 hybrid_clock.cc:648] HybridClock initialized: now 1790598906034929 us; error 35 us; skew 500 ppm
Sep 28 12:35:06 dist-test-slave-g31s krb5kdc[29766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598906, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:06.037379 29779 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:35:06.037788 29779 webserver.cc:460] Webserver started at http://127.8.13.62:41405/ using document root <none> and password file <none>
I20260928 12:35:06.038005 29779 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:06.038045 29779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:06.038147 29779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:06.039182 29779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "515043e90552479cbb6fc8ba079f189d"
format_stamp: "Formatted at 2026-09-28 12:35:06 on dist-test-slave-g31s"
server_key: "5558aa908eb1389e6aab0a5573976dbb"
server_key_iv: "011b3234f065357f597b51b7d4f45120"
server_key_version: "encryptionkey@0"
I20260928 12:35:06.039476 29779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "515043e90552479cbb6fc8ba079f189d"
format_stamp: "Formatted at 2026-09-28 12:35:06 on dist-test-slave-g31s"
server_key: "5558aa908eb1389e6aab0a5573976dbb"
server_key_iv: "011b3234f065357f597b51b7d4f45120"
server_key_version: "encryptionkey@0"
I20260928 12:35:06.040721 29779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:35:06.041395 29794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:06.041527 29779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:35:06.041606 29779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "515043e90552479cbb6fc8ba079f189d"
format_stamp: "Formatted at 2026-09-28 12:35:06 on dist-test-slave-g31s"
server_key: "5558aa908eb1389e6aab0a5573976dbb"
server_key_iv: "011b3234f065357f597b51b7d4f45120"
server_key_version: "encryptionkey@0"
I20260928 12:35:06.041675 29779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:06.057461 29779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:06.059520 29779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:06.059656 29779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:06.063530 29779 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:35261
I20260928 12:35:06.063575 29846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:35261 every 8 connection(s)
I20260928 12:35:06.063911 29779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:06.064771 29847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:06.067214 29847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d: Bootstrap starting.
I20260928 12:35:06.067773 29847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:06.067994 29847 log.cc:824] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:06.068652 29847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d: No bootstrap required, opened a new log
I20260928 12:35:06.069166 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 29779
I20260928 12:35:06.069274 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:06.069465 8244 external_mini_cluster.cc:1428] Setting key 7f7280baa49b12b44081207f59bd4791
I20260928 12:35:06.070761 29847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515043e90552479cbb6fc8ba079f189d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35261 } }
I20260928 12:35:06.070926 29847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:35:06.070966 29847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515043e90552479cbb6fc8ba079f189d, State: Initialized, Role: FOLLOWER
I20260928 12:35:06.071111 29847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515043e90552479cbb6fc8ba079f189d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35261 } }
I20260928 12:35:06.071192 29847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:35:06.071228 29847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:35:06.071285 29847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:06.072297 29847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515043e90552479cbb6fc8ba079f189d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35261 } }
I20260928 12:35:06.072405 29847 leader_election.cc:302] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515043e90552479cbb6fc8ba079f189d; no voters:
I20260928 12:35:06.072575 29847 leader_election.cc:288] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:35:06.072618 29852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:06.072762 29852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [term 1 LEADER]: Becoming Leader. State: Replica: 515043e90552479cbb6fc8ba079f189d, State: Running, Role: LEADER
I20260928 12:35:06.072852 29847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:06.072868 29852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515043e90552479cbb6fc8ba079f189d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35261 } }
I20260928 12:35:06.073199 29852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515043e90552479cbb6fc8ba079f189d. Latest consensus state: current_term: 1 leader_uuid: "515043e90552479cbb6fc8ba079f189d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515043e90552479cbb6fc8ba079f189d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35261 } } }
I20260928 12:35:06.073271 29852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:06.073410 29853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "515043e90552479cbb6fc8ba079f189d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515043e90552479cbb6fc8ba079f189d" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 35261 } } }
I20260928 12:35:06.073472 29853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:06.073588 29857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:06.074122 29857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:06.075765 29857 catalog_manager.cc:1347] Generated new cluster ID: 89e8e701164a4e6cba281f1ff22602b6
I20260928 12:35:06.075824 29857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:35:06 dist-test-slave-g31s krb5kdc[29766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:35:06.080245 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:06.070004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35101 (local address 127.8.13.62:35261)
0928 12:35:06.070177 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:06.070185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:06.070211 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:35:06.070721 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:06.070728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:06.070774 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:06.070883 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:06.071653 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:06.072647 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:06.073690 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:06.074363 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:06.074555 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:06.077044 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:06.077050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:06.077050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:06.077064 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:06.078697 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:06.079078 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:06.079081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:06.079082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:06.079196 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:06.079394 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:06.079394 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:06.079395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:06.079664 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:06.079818 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:06.079936 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 12:35:06.080038 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260928 12:35:06.081161 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:35:06.091547 29857 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:06.092121 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:06.086188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52939 (local address 127.8.13.62:35261)
0928 12:35:06.086229 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:06.086231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:06.086360 (+ 129us) server_negotiation.cc:407] Connection header received
0928 12:35:06.086422 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:06.086423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:06.086433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:06.086477 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:06.087005 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:06.087503 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:06.088156 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:06.088366 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:06.088420 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:06.089577 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:06.089578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:06.089579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:06.089586 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:06.090846 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:06.091245 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:06.091246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:06.091247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:06.091317 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:06.091598 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:06.091599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:06.091600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:06.091731 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:06.091858 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:06.091971 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 12:35:06.092032 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:35:06.092255 29857 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:06.098898 29857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 515043e90552479cbb6fc8ba079f189d: Generated new TSK 0
I20260928 12:35:06.099107 29857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:35:06.102155 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 29779
W20260928 12:35:06.105865 29764 connection.cc:535] client connection to 127.8.13.62:35261 recv error: Network error: failed to read from TLS socket (remote: 127.8.13.62:35261): Connection reset by peer (error 104)
2026-09-28T12:35:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:35:06 dist-test-slave-g31s krb5kdc[29878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:35:06 dist-test-slave-g31s krb5kdc[29878](info): set up 2 sockets
Sep 28 12:35:06 dist-test-slave-g31s krb5kdc[29878](info): commencing operation
krb5kdc: starting...
W20260928 12:35:07.045914 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:07 dist-test-slave-g31s krb5kdc[29878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:35:07.156925 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:41523
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:41523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:07.237653 29891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:07.237910 29891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:07.237938 29891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:07.239118 29891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:07.239162 29891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:07.239176 29891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:07.239187 29891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:07.239197 29891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:07.240670 29891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:41523
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:41523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:07.240875 29891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:07.241063 29891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:07.246004 29896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:07.246021 29897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:07.246032 29891 server_base.cc:1034] running on GCE node
W20260928 12:35:07.246021 29899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:07.246387 29891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:07.246667 29891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:07.247808 29891 hybrid_clock.cc:648] HybridClock initialized: now 1790598907247799 us; error 28 us; skew 500 ppm
Sep 28 12:35:07 dist-test-slave-g31s krb5kdc[29878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598907, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:07.250336 29891 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:35:07.250809 29891 webserver.cc:460] Webserver started at http://127.8.13.62:33561/ using document root <none> and password file <none>
I20260928 12:35:07.251034 29891 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:07.251080 29891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:07.251217 29891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:07.252345 29891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "75f76733b1eb49e292e7afd817e48bd3"
format_stamp: "Formatted at 2026-09-28 12:35:07 on dist-test-slave-g31s"
server_key: "20a1b4dd029c4a713be33472958b2131"
server_key_iv: "b7ef4915cdefdf9e62e18ddbb072f491"
server_key_version: "encryptionkey@0"
I20260928 12:35:07.252642 29891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "75f76733b1eb49e292e7afd817e48bd3"
format_stamp: "Formatted at 2026-09-28 12:35:07 on dist-test-slave-g31s"
server_key: "20a1b4dd029c4a713be33472958b2131"
server_key_iv: "b7ef4915cdefdf9e62e18ddbb072f491"
server_key_version: "encryptionkey@0"
I20260928 12:35:07.253914 29891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:35:07.254635 29906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:07.254778 29891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:07.254858 29891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "75f76733b1eb49e292e7afd817e48bd3"
format_stamp: "Formatted at 2026-09-28 12:35:07 on dist-test-slave-g31s"
server_key: "20a1b4dd029c4a713be33472958b2131"
server_key_iv: "b7ef4915cdefdf9e62e18ddbb072f491"
server_key_version: "encryptionkey@0"
I20260928 12:35:07.254928 29891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:07.286489 29891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:07.288319 29891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:07.288445 29891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:07.292450 29891 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:41523
I20260928 12:35:07.292488 29958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:41523 every 8 connection(s)
I20260928 12:35:07.292833 29891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:07.293732 29959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:07.296134 29959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3: Bootstrap starting.
I20260928 12:35:07.296671 29959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:07.296901 29959 log.cc:824] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:07.297556 29959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3: No bootstrap required, opened a new log
I20260928 12:35:07.299219 29959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f76733b1eb49e292e7afd817e48bd3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41523 } }
I20260928 12:35:07.299372 29959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:35:07.299393 29959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f76733b1eb49e292e7afd817e48bd3, State: Initialized, Role: FOLLOWER
I20260928 12:35:07.299515 29959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f76733b1eb49e292e7afd817e48bd3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41523 } }
I20260928 12:35:07.299592 29959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:35:07.299618 29959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:35:07.299651 29959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:07.300482 29959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f76733b1eb49e292e7afd817e48bd3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41523 } }
I20260928 12:35:07.300575 29959 leader_election.cc:302] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75f76733b1eb49e292e7afd817e48bd3; no voters:
I20260928 12:35:07.300704 29959 leader_election.cc:288] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:35:07.300755 29962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:07.300885 29962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [term 1 LEADER]: Becoming Leader. State: Replica: 75f76733b1eb49e292e7afd817e48bd3, State: Running, Role: LEADER
I20260928 12:35:07.300966 29959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:07.301002 29962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f76733b1eb49e292e7afd817e48bd3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41523 } }
I20260928 12:35:07.301401 29964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75f76733b1eb49e292e7afd817e48bd3. Latest consensus state: current_term: 1 leader_uuid: "75f76733b1eb49e292e7afd817e48bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f76733b1eb49e292e7afd817e48bd3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41523 } } }
I20260928 12:35:07.301491 29964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:07.301380 29963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75f76733b1eb49e292e7afd817e48bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f76733b1eb49e292e7afd817e48bd3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 41523 } } }
I20260928 12:35:07.301964 29963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:07.301823 29970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:07.302148 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 29891
I20260928 12:35:07.302273 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:07.302449 8244 external_mini_cluster.cc:1428] Setting key 0a8b9ef728b6605b11c91e58bfa10b1b
I20260928 12:35:07.302606 29970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:07.304332 29970 catalog_manager.cc:1347] Generated new cluster ID: 6c2d1c0e8c8d48b9b8b9ee915578e850
I20260928 12:35:07.304385 29970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:35:07.307794 29981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:07.303009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56369 (local address 127.8.13.62:41523)
0928 12:35:07.303371 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:07.303381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:07.303411 (+ 30us) server_negotiation.cc:407] Connection header received
0928 12:35:07.303541 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:07.303548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:07.303603 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:07.303736 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:07.304045 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:07.305066 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:07.305854 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:07.306346 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:07.306450 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:07.307140 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:07.307144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:07.307144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:07.307317 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:07.307347 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:07.307358 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:07.307424 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:07.307515 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 12:35:07.307631 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":63,"threads_started":1}
I20260928 12:35:07.308635 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:35:07.316546 29981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:07.309888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49279 (local address 127.8.13.62:41523)
0928 12:35:07.309921 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:07.309923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:07.314005 (+ 4082us) server_negotiation.cc:407] Connection header received
0928 12:35:07.314100 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:07.314101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:07.314112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:07.314162 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:07.314289 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:07.314790 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:07.315449 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:07.315666 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:07.315718 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:07.316321 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:07.316323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:07.316323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:07.316328 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:07.316342 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:07.316347 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:07.316388 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:07.316453 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 12:35:07.316466 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:35:07.324817 29970 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:07.325333 29970 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:07.333899 29970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75f76733b1eb49e292e7afd817e48bd3: Generated new TSK 0
I20260928 12:35:07.334098 29970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:35:07.340456 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 29891
W20260928 12:35:07.344532 29876 connection.cc:535] client connection to 127.8.13.62:41523 recv error: Network error: failed to read from TLS socket (remote: 127.8.13.62:41523): Connection reset by peer (error 104)
2026-09-28T12:35:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1238 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:35:07 dist-test-slave-g31s krb5kdc[29990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:35:07 dist-test-slave-g31s krb5kdc[29990](info): set up 2 sockets
Sep 28 12:35:07 dist-test-slave-g31s krb5kdc[29990](info): commencing operation
krb5kdc: starting...
W20260928 12:35:08.289722 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:08 dist-test-slave-g31s krb5kdc[29990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:35:08.402228 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:39139
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:39139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:08.482549 30003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:08.482795 30003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:08.482820 30003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:08.484079 30003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:08.484130 30003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:08.484144 30003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:08.484158 30003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:08.484190 30003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:08.485790 30003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:39139
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:39139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:08.486066 30003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:08.486296 30003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:08.490931 30009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:08.490952 30008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:08.490953 30011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:08.491161 30003 server_base.cc:1034] running on GCE node
I20260928 12:35:08.491423 30003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:08.491736 30003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:08.492901 30003 hybrid_clock.cc:648] HybridClock initialized: now 1790598908492883 us; error 30 us; skew 500 ppm
Sep 28 12:35:08 dist-test-slave-g31s krb5kdc[29990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598908, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:08.495429 30003 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:35:08.495882 30003 webserver.cc:460] Webserver started at http://127.8.13.62:44373/ using document root <none> and password file <none>
I20260928 12:35:08.496083 30003 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:08.496130 30003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:08.496258 30003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:08.497316 30003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "ca15af90b1944ae9966da51906660091"
format_stamp: "Formatted at 2026-09-28 12:35:08 on dist-test-slave-g31s"
server_key: "c23d3788ea1b1ca039e2d74dff53603f"
server_key_iv: "19c7ddfeded86167ff3bc76bcf981c38"
server_key_version: "encryptionkey@0"
I20260928 12:35:08.497645 30003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "ca15af90b1944ae9966da51906660091"
format_stamp: "Formatted at 2026-09-28 12:35:08 on dist-test-slave-g31s"
server_key: "c23d3788ea1b1ca039e2d74dff53603f"
server_key_iv: "19c7ddfeded86167ff3bc76bcf981c38"
server_key_version: "encryptionkey@0"
I20260928 12:35:08.498971 30003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:35:08.499670 30018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:08.499826 30003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:35:08.499909 30003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "ca15af90b1944ae9966da51906660091"
format_stamp: "Formatted at 2026-09-28 12:35:08 on dist-test-slave-g31s"
server_key: "c23d3788ea1b1ca039e2d74dff53603f"
server_key_iv: "19c7ddfeded86167ff3bc76bcf981c38"
server_key_version: "encryptionkey@0"
I20260928 12:35:08.499979 30003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:08.503345 30003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:08.505121 30003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:08.505267 30003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:08.509058 30003 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:39139
I20260928 12:35:08.509079 30070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:39139 every 8 connection(s)
I20260928 12:35:08.509411 30003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:08.510386 30071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:08.512625 30071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091: Bootstrap starting.
I20260928 12:35:08.513161 30071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:08.513397 30071 log.cc:824] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:08.514052 30071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091: No bootstrap required, opened a new log
I20260928 12:35:08.515666 30071 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca15af90b1944ae9966da51906660091" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39139 } }
I20260928 12:35:08.515799 30071 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:35:08.515821 30071 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca15af90b1944ae9966da51906660091, State: Initialized, Role: FOLLOWER
I20260928 12:35:08.515947 30071 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca15af90b1944ae9966da51906660091" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39139 } }
I20260928 12:35:08.516003 30071 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:35:08.516024 30071 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:35:08.516077 30071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:08.516752 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 30003
I20260928 12:35:08.516884 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:08.516968 30071 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca15af90b1944ae9966da51906660091" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39139 } }
I20260928 12:35:08.517067 8244 external_mini_cluster.cc:1428] Setting key e8171da2c031368a13c8fd67d5794a15
I20260928 12:35:08.517078 30071 leader_election.cc:302] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca15af90b1944ae9966da51906660091; no voters:
I20260928 12:35:08.517232 30071 leader_election.cc:288] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:35:08.517274 30074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:08.517467 30074 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [term 1 LEADER]: Becoming Leader. State: Replica: ca15af90b1944ae9966da51906660091, State: Running, Role: LEADER
I20260928 12:35:08.517473 30071 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:08.517642 30074 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca15af90b1944ae9966da51906660091" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39139 } }
I20260928 12:35:08.518000 30075 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca15af90b1944ae9966da51906660091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca15af90b1944ae9966da51906660091" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39139 } } }
I20260928 12:35:08.518013 30076 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca15af90b1944ae9966da51906660091. Latest consensus state: current_term: 1 leader_uuid: "ca15af90b1944ae9966da51906660091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca15af90b1944ae9966da51906660091" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39139 } } }
I20260928 12:35:08.518085 30075 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:08.518103 30076 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091 [sys.catalog]: This master's current role is: LEADER
W20260928 12:35:08.519331 30092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:35:08.519383 30092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:35:08.519460 30085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:08.519636 30085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:08.520304 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:08.517655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41455 (local address 127.8.13.62:39139)
0928 12:35:08.517883 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:08.517892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:08.517924 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:35:08.518039 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:08.518046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:08.518087 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:08.518488 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:08.518949 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:08.518952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:08.518952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:08.518956 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:08.518994 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:08.519007 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:08.519205 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:08.519237 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 12:35:08.519278 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":65,"threads_started":1}
I20260928 12:35:08.521106 30085 catalog_manager.cc:1347] Generated new cluster ID: dc3cadc2a82140a88c7c0a7fad73bd4a
I20260928 12:35:08.521153 30085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:35:08.521142 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:35:08.523658 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:08.522605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35097 (local address 127.8.13.62:39139)
0928 12:35:08.522636 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:08.522638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:08.522769 (+ 131us) server_negotiation.cc:407] Connection header received
0928 12:35:08.522834 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:08.522835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:08.522839 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:08.522894 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:08.523476 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:08.523477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:08.523478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:08.523481 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:08.523495 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:08.523498 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:08.523539 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:08.523590 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 12:35:08.523602 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:35:08.529034 30085 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:08.529217 30085 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:08.539304 30085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca15af90b1944ae9966da51906660091: Generated new TSK 0
I20260928 12:35:08.539538 30085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:35:08.543115 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 30003
2026-09-28T12:35:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:35:08 dist-test-slave-g31s krb5kdc[30102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:35:08 dist-test-slave-g31s krb5kdc[30102](info): set up 2 sockets
Sep 28 12:35:08 dist-test-slave-g31s krb5kdc[30102](info): commencing operation
krb5kdc: starting...
W20260928 12:35:09.489607 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:09 dist-test-slave-g31s krb5kdc[30102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:35:09.604882 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:38031
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:42369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:38031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:09.684898 30115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:09.685213 30115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:09.685269 30115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:09.686520 30115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:09.686568 30115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:09.686582 30115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:09.686594 30115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:09.686604 30115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:09.688148 30115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:42369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:38031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:38031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:09.688369 30115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:09.688553 30115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:09.693156 30120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:09.693281 30115 server_base.cc:1034] running on GCE node
W20260928 12:35:09.693163 30121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:09.693388 30123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:09.693667 30115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:09.694027 30115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:09.695191 30115 hybrid_clock.cc:648] HybridClock initialized: now 1790598909695170 us; error 38 us; skew 500 ppm
Sep 28 12:35:09 dist-test-slave-g31s krb5kdc[30102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598909, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:09.697708 30115 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:35:09.698200 30115 webserver.cc:460] Webserver started at http://127.8.13.62:34751/ using document root <none> and password file <none>
I20260928 12:35:09.698410 30115 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:09.698452 30115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:09.698583 30115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:09.699648 30115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "83dbb07b7ca0457e81c5bf1f388bc289"
format_stamp: "Formatted at 2026-09-28 12:35:09 on dist-test-slave-g31s"
server_key: "4d76036a8adcf54f282b3bb88e79ccae"
server_key_iv: "bfa4e04150ef3e952d637415f2e87e6c"
server_key_version: "encryptionkey@0"
I20260928 12:35:09.699978 30115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "83dbb07b7ca0457e81c5bf1f388bc289"
format_stamp: "Formatted at 2026-09-28 12:35:09 on dist-test-slave-g31s"
server_key: "4d76036a8adcf54f282b3bb88e79ccae"
server_key_iv: "bfa4e04150ef3e952d637415f2e87e6c"
server_key_version: "encryptionkey@0"
I20260928 12:35:09.701252 30115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:35:09.701979 30130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:09.702127 30115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:35:09.702210 30115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "83dbb07b7ca0457e81c5bf1f388bc289"
format_stamp: "Formatted at 2026-09-28 12:35:09 on dist-test-slave-g31s"
server_key: "4d76036a8adcf54f282b3bb88e79ccae"
server_key_iv: "bfa4e04150ef3e952d637415f2e87e6c"
server_key_version: "encryptionkey@0"
I20260928 12:35:09.702278 30115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:09.713877 30115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:09.715723 30115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:09.715842 30115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:09.719702 30115 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:38031
I20260928 12:35:09.719738 30182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:38031 every 8 connection(s)
I20260928 12:35:09.720062 30115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:09.720906 30183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:09.723176 30183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289: Bootstrap starting.
I20260928 12:35:09.723716 30183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:09.723973 30183 log.cc:824] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:09.724608 30183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289: No bootstrap required, opened a new log
I20260928 12:35:09.726218 30183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dbb07b7ca0457e81c5bf1f388bc289" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38031 } }
I20260928 12:35:09.726365 30183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:35:09.726401 30183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83dbb07b7ca0457e81c5bf1f388bc289, State: Initialized, Role: FOLLOWER
I20260928 12:35:09.726558 30183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dbb07b7ca0457e81c5bf1f388bc289" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38031 } }
I20260928 12:35:09.726619 30183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:35:09.726642 30183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:35:09.726689 30183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:09.727519 30183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dbb07b7ca0457e81c5bf1f388bc289" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38031 } }
I20260928 12:35:09.727612 30183 leader_election.cc:302] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83dbb07b7ca0457e81c5bf1f388bc289; no voters:
I20260928 12:35:09.727782 30183 leader_election.cc:288] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:35:09.727816 30186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:09.727946 30186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [term 1 LEADER]: Becoming Leader. State: Replica: 83dbb07b7ca0457e81c5bf1f388bc289, State: Running, Role: LEADER
I20260928 12:35:09.728026 30183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:09.728032 30186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dbb07b7ca0457e81c5bf1f388bc289" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38031 } }
I20260928 12:35:09.728394 30187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83dbb07b7ca0457e81c5bf1f388bc289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dbb07b7ca0457e81c5bf1f388bc289" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38031 } } }
I20260928 12:35:09.728401 30188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83dbb07b7ca0457e81c5bf1f388bc289. Latest consensus state: current_term: 1 leader_uuid: "83dbb07b7ca0457e81c5bf1f388bc289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dbb07b7ca0457e81c5bf1f388bc289" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38031 } } }
I20260928 12:35:09.728554 30188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:09.728888 30187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289 [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:09.729040 30195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:09.729547 30195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:09.729576 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 30115
I20260928 12:35:09.729686 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:09.729904 8244 external_mini_cluster.cc:1428] Setting key 675c2940a0f6df6502011192a453e684
I20260928 12:35:09.731218 30195 catalog_manager.cc:1347] Generated new cluster ID: 378f5e574f814975b4e0f45c5f564c33
I20260928 12:35:09.731268 30195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:35:09 dist-test-slave-g31s krb5kdc[30102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:35:09.741564 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:09.730341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48951 (local address 127.8.13.62:38031)
0928 12:35:09.730488 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:09.730497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:09.730615 (+ 118us) server_negotiation.cc:407] Connection header received
0928 12:35:09.730746 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:09.730754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:09.730826 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:09.730956 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:09.731690 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:09.733133 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:09.734210 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:09.734849 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:09.734975 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:09.737858 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:09.737863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:09.737864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:09.737878 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:09.739864 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:09.740321 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:09.740323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:09.740324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:09.740436 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:09.740615 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:09.740615 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:09.740615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:09.740933 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:09.741054 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:09.741170 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 12:35:09.741318 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260928 12:35:09.742739 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:35:09.752861 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:09.744119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52731 (local address 127.8.13.62:38031)
0928 12:35:09.744154 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:09.744156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:09.745947 (+ 1791us) server_negotiation.cc:407] Connection header received
0928 12:35:09.746040 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:09.746041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:09.746053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:09.746114 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:09.746723 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:09.747496 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:09.748504 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:09.748837 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:09.748904 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:09.750263 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:09.750265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:09.750265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:09.750272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:09.751788 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:09.752129 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:09.752130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:09.752131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:09.752197 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:09.752359 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:09.752360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:09.752360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:09.752493 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:09.752588 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:09.752672 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 12:35:09.752756 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 12:35:09.757145 30195 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:09.757664 30195 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:09.761986 30195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83dbb07b7ca0457e81c5bf1f388bc289: Generated new TSK 0
I20260928 12:35:09.762193 30195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:35:09.765173 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 30115
2026-09-28T12:35:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:35:09 dist-test-slave-g31s krb5kdc[30214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:35:09 dist-test-slave-g31s krb5kdc[30214](info): set up 2 sockets
Sep 28 12:35:09 dist-test-slave-g31s krb5kdc[30214](info): commencing operation
krb5kdc: starting...
W20260928 12:35:10.729671 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:10 dist-test-slave-g31s krb5kdc[30214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:35:10.845012 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:38153
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:38153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:10.926152 30227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:10.926355 30227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:10.926378 30227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:10.927506 30227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:10.927547 30227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:10.927561 30227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:10.927572 30227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:10.927595 30227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:10.929055 30227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:38153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:38153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:10.929286 30227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:10.929488 30227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:10.934012 30232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:10.934036 30233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:10.934255 30227 server_base.cc:1034] running on GCE node
W20260928 12:35:10.934013 30235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:10.934502 30227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:10.934762 30227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:10.935887 30227 hybrid_clock.cc:648] HybridClock initialized: now 1790598910935879 us; error 27 us; skew 500 ppm
Sep 28 12:35:10 dist-test-slave-g31s krb5kdc[30214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598910, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:10.938191 30227 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:35:10.938553 30227 webserver.cc:460] Webserver started at http://127.8.13.62:41331/ using document root <none> and password file <none>
I20260928 12:35:10.938726 30227 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:10.938764 30227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:10.938848 30227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:10.939827 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "349799cae9684c6ea860a19189a9d35a"
format_stamp: "Formatted at 2026-09-28 12:35:10 on dist-test-slave-g31s"
server_key: "fd87ce2ba638ea3d7d0edec2df028e47"
server_key_iv: "f291398e5c7b47e087a3acc0b4676b6a"
server_key_version: "encryptionkey@0"
I20260928 12:35:10.940104 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "349799cae9684c6ea860a19189a9d35a"
format_stamp: "Formatted at 2026-09-28 12:35:10 on dist-test-slave-g31s"
server_key: "fd87ce2ba638ea3d7d0edec2df028e47"
server_key_iv: "f291398e5c7b47e087a3acc0b4676b6a"
server_key_version: "encryptionkey@0"
I20260928 12:35:10.941267 30227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:35:10.941913 30242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:10.942050 30227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:10.942132 30227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "349799cae9684c6ea860a19189a9d35a"
format_stamp: "Formatted at 2026-09-28 12:35:10 on dist-test-slave-g31s"
server_key: "fd87ce2ba638ea3d7d0edec2df028e47"
server_key_iv: "f291398e5c7b47e087a3acc0b4676b6a"
server_key_version: "encryptionkey@0"
I20260928 12:35:10.942188 30227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:10.957638 30227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:10.959455 30227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:10.959561 30227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:10.963091 30227 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:38153
I20260928 12:35:10.963165 30294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:38153 every 8 connection(s)
I20260928 12:35:10.963464 30227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:10.964309 30295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:10.966665 30295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a: Bootstrap starting.
I20260928 12:35:10.967211 30295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:10.967453 30295 log.cc:824] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:10.968065 30295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a: No bootstrap required, opened a new log
I20260928 12:35:10.969766 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 30227
I20260928 12:35:10.969909 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:10.969951 30295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349799cae9684c6ea860a19189a9d35a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38153 } }
I20260928 12:35:10.970093 8244 external_mini_cluster.cc:1428] Setting key d7ade4018c12c0175724f4e8f528a46d
I20260928 12:35:10.970134 30295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:35:10.970168 30295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349799cae9684c6ea860a19189a9d35a, State: Initialized, Role: FOLLOWER
I20260928 12:35:10.970309 30295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349799cae9684c6ea860a19189a9d35a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38153 } }
I20260928 12:35:10.970391 30295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:35:10.970422 30295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:35:10.970484 30295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:10.971349 30295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349799cae9684c6ea860a19189a9d35a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38153 } }
I20260928 12:35:10.971442 30295 leader_election.cc:302] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 349799cae9684c6ea860a19189a9d35a; no voters:
I20260928 12:35:10.971587 30295 leader_election.cc:288] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:35:10.971628 30300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:10.971781 30300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [term 1 LEADER]: Becoming Leader. State: Replica: 349799cae9684c6ea860a19189a9d35a, State: Running, Role: LEADER
I20260928 12:35:10.971854 30295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:10.971926 30300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349799cae9684c6ea860a19189a9d35a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38153 } }
I20260928 12:35:10.972313 30300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 349799cae9684c6ea860a19189a9d35a. Latest consensus state: current_term: 1 leader_uuid: "349799cae9684c6ea860a19189a9d35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349799cae9684c6ea860a19189a9d35a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38153 } } }
I20260928 12:35:10.972355 30302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "349799cae9684c6ea860a19189a9d35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349799cae9684c6ea860a19189a9d35a" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 38153 } } }
I20260928 12:35:10.972453 30302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:10.972440 30300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a [sys.catalog]: This master's current role is: LEADER
W20260928 12:35:10.973627 30316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:35:10.973682 30316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:35:10.973729 30309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:10.973909 30309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:10.975279 30309 catalog_manager.cc:1347] Generated new cluster ID: 6b4c0f3bd5874d6f9a344539729ad0fd
I20260928 12:35:10.975332 30309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:35:10.976246 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:10.970561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49667 (local address 127.8.13.62:38153)
0928 12:35:10.970760 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:10.970770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:10.970801 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:35:10.970911 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:10.970917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:10.970959 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:10.971069 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:10.971303 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:10.972500 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:10.973611 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:10.974849 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:10.974970 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:10.975640 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:10.975643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:10.975643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:10.975823 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:10.975853 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:10.975863 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:10.975914 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:10.975987 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 12:35:10.976047 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260928 12:35:10.977041 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:35:10.980834 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:10.978276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58499 (local address 127.8.13.62:38153)
0928 12:35:10.978309 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:10.978311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:10.978417 (+ 106us) server_negotiation.cc:407] Connection header received
0928 12:35:10.978467 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:10.978468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:10.978488 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:10.978539 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:10.978657 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:10.979155 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:10.979823 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:10.980032 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:10.980085 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:10.980623 (+ 538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:10.980625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:10.980625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:10.980629 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:10.980643 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:10.980647 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:10.980687 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:10.980740 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 12:35:10.980764 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":16}
I20260928 12:35:10.998934 30309 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:11.000221 30309 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:11.017427 30309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 349799cae9684c6ea860a19189a9d35a: Generated new TSK 0
I20260928 12:35:11.017647 30309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:35:11.024300 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 30227
2026-09-28T12:35:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1258 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:35:11 dist-test-slave-g31s krb5kdc[30326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:35:11 dist-test-slave-g31s krb5kdc[30326](info): set up 2 sockets
Sep 28 12:35:11 dist-test-slave-g31s krb5kdc[30326](info): commencing operation
krb5kdc: starting...
W20260928 12:35:11.973708 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:12 dist-test-slave-g31s krb5kdc[30326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:35:12.090730 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:39289
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:39289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:12.169111 30339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:12.169355 30339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:12.169390 30339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:12.170673 30339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:12.170725 30339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:12.170745 30339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:12.170763 30339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:12.170776 30339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:12.172456 30339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:39289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:39289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:12.172737 30339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:12.172951 30339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:12.177726 30347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:12.177726 30344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:12.177942 30339 server_base.cc:1034] running on GCE node
W20260928 12:35:12.177879 30345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:12.178267 30339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:12.178562 30339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:12.179724 30339 hybrid_clock.cc:648] HybridClock initialized: now 1790598912179705 us; error 28 us; skew 500 ppm
Sep 28 12:35:12 dist-test-slave-g31s krb5kdc[30326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598912, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:12.183037 30339 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:35:12.183629 30339 webserver.cc:460] Webserver started at http://127.8.13.62:46245/ using document root <none> and password file <none>
I20260928 12:35:12.183905 30339 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:12.183970 30339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:12.184163 30339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:12.185673 30339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "6d20e025e7de4d1c8d43f98f758e7ffb"
format_stamp: "Formatted at 2026-09-28 12:35:12 on dist-test-slave-g31s"
server_key: "6e3358f8055f51cf0dea53a3d093de41"
server_key_iv: "252a79372d888644e53b686f8417dd0d"
server_key_version: "encryptionkey@0"
I20260928 12:35:12.186120 30339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "6d20e025e7de4d1c8d43f98f758e7ffb"
format_stamp: "Formatted at 2026-09-28 12:35:12 on dist-test-slave-g31s"
server_key: "6e3358f8055f51cf0dea53a3d093de41"
server_key_iv: "252a79372d888644e53b686f8417dd0d"
server_key_version: "encryptionkey@0"
I20260928 12:35:12.187410 30339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:35:12.188110 30354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:12.188256 30339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:35:12.188341 30339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "6d20e025e7de4d1c8d43f98f758e7ffb"
format_stamp: "Formatted at 2026-09-28 12:35:12 on dist-test-slave-g31s"
server_key: "6e3358f8055f51cf0dea53a3d093de41"
server_key_iv: "252a79372d888644e53b686f8417dd0d"
server_key_version: "encryptionkey@0"
I20260928 12:35:12.188400 30339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:12.191859 30339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:12.193984 30339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:12.194097 30339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:12.197831 30339 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:39289
I20260928 12:35:12.197877 30406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:39289 every 8 connection(s)
I20260928 12:35:12.198189 30339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:12.199148 30407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:12.201440 30407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb: Bootstrap starting.
I20260928 12:35:12.202072 30407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:12.202328 30407 log.cc:824] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:12.203035 30407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb: No bootstrap required, opened a new log
I20260928 12:35:12.204634 30407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20e025e7de4d1c8d43f98f758e7ffb" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39289 } }
I20260928 12:35:12.204802 30407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:35:12.204833 30407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20e025e7de4d1c8d43f98f758e7ffb, State: Initialized, Role: FOLLOWER
I20260928 12:35:12.204964 30407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20e025e7de4d1c8d43f98f758e7ffb" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39289 } }
I20260928 12:35:12.205026 30407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:35:12.205056 30407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:35:12.205102 30407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:12.205256 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 30339
I20260928 12:35:12.205355 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:12.205525 8244 external_mini_cluster.cc:1428] Setting key 441972d22f757be527c07989fab9f46b
I20260928 12:35:12.206106 30407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20e025e7de4d1c8d43f98f758e7ffb" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39289 } }
I20260928 12:35:12.206210 30407 leader_election.cc:302] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d20e025e7de4d1c8d43f98f758e7ffb; no voters:
I20260928 12:35:12.206357 30407 leader_election.cc:288] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:35:12.206408 30412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:12.206578 30412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [term 1 LEADER]: Becoming Leader. State: Replica: 6d20e025e7de4d1c8d43f98f758e7ffb, State: Running, Role: LEADER
I20260928 12:35:12.206595 30407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:12.207327 30412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20e025e7de4d1c8d43f98f758e7ffb" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39289 } }
I20260928 12:35:12.207778 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:12.205966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57199 (local address 127.8.13.62:39289)
0928 12:35:12.206155 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:12.206164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:12.206199 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:35:12.206318 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:12.206325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:12.206357 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:12.206508 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:12.207222 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:12.207225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:12.207225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:12.207230 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:12.207277 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:12.207292 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:12.207361 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:12.207497 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 12:35:12.207542 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260928 12:35:12.207947 30413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d20e025e7de4d1c8d43f98f758e7ffb. Latest consensus state: current_term: 1 leader_uuid: "6d20e025e7de4d1c8d43f98f758e7ffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20e025e7de4d1c8d43f98f758e7ffb" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39289 } } }
I20260928 12:35:12.208071 30414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d20e025e7de4d1c8d43f98f758e7ffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20e025e7de4d1c8d43f98f758e7ffb" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 39289 } } }
I20260928 12:35:12.208171 30414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:12.208168 30413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:12.208397 30428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:12.208928 30428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:12.208971 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:35:12.210307 30428 catalog_manager.cc:1347] Generated new cluster ID: e6d7fc175b1a4f45a859fff9087dc538
I20260928 12:35:12.210358 30428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:35:12.211581 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:12.210451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48119 (local address 127.8.13.62:39289)
0928 12:35:12.210484 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:12.210485 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:12.210637 (+ 152us) server_negotiation.cc:407] Connection header received
0928 12:35:12.210730 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:12.210731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:12.210736 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:12.210761 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:12.211362 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:12.211363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:12.211363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:35:12.211368 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:12.211380 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:35:12.211384 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:12.211414 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:12.211512 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 12:35:12.211525 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:35:12.224121 30428 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:12.224288 30428 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:12.232746 30428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d20e025e7de4d1c8d43f98f758e7ffb: Generated new TSK 0
I20260928 12:35:12.232947 30428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:35:12.242954 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 30339
2026-09-28T12:35:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1218 ms)
[----------] 18 tests from AuthTokenIssuingTest (22018 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:35:12 dist-test-slave-g31s krb5kdc[30438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:35:12 dist-test-slave-g31s krb5kdc[30438](info): set up 2 sockets
Sep 28 12:35:12 dist-test-slave-g31s krb5kdc[30438](info): commencing operation
krb5kdc: starting...
W20260928 12:35:13.190584 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:35:13.312314 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:34227
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:34227,127.8.13.61:37105,127.8.13.60:44493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:13.392511 30451 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:35:13.392753 30451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:13.392771 30451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:13.392793 30451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:13.394093 30451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:13.394140 30451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:13.394152 30451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:13.394164 30451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:13.394176 30451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:13.395738 30451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:34227,127.8.13.61:37105,127.8.13.60:44493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:34227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:13.395988 30451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:13.396203 30451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:13.400985 30459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:13.400988 30456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:13.400985 30457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:13.401189 30451 server_base.cc:1034] running on GCE node
I20260928 12:35:13.401469 30451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:13.401769 30451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:13.402937 30451 hybrid_clock.cc:648] HybridClock initialized: now 1790598913402925 us; error 29 us; skew 500 ppm
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:13.405239 30451 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:35:13.405674 30451 webserver.cc:460] Webserver started at http://127.8.13.62:39757/ using document root <none> and password file <none>
I20260928 12:35:13.405933 30451 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:13.405980 30451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:13.406088 30451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:13.407202 30451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "a32e17ce622f450f83148bba65ced1f9"
format_stamp: "Formatted at 2026-09-28 12:35:13 on dist-test-slave-g31s"
server_key: "1d3dc3bf16c36cf86d746363e8e5bdfd"
server_key_iv: "4869215e69979cf39db7bd213c8b97b8"
server_key_version: "encryptionkey@0"
I20260928 12:35:13.407532 30451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "a32e17ce622f450f83148bba65ced1f9"
format_stamp: "Formatted at 2026-09-28 12:35:13 on dist-test-slave-g31s"
server_key: "1d3dc3bf16c36cf86d746363e8e5bdfd"
server_key_iv: "4869215e69979cf39db7bd213c8b97b8"
server_key_version: "encryptionkey@0"
I20260928 12:35:13.408757 30451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:35:13.409404 30466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:13.409577 30451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:35:13.409663 30451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "a32e17ce622f450f83148bba65ced1f9"
format_stamp: "Formatted at 2026-09-28 12:35:13 on dist-test-slave-g31s"
server_key: "1d3dc3bf16c36cf86d746363e8e5bdfd"
server_key_iv: "4869215e69979cf39db7bd213c8b97b8"
server_key_version: "encryptionkey@0"
I20260928 12:35:13.409719 30451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:13.426838 30451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:13.428655 30451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:13.428764 30451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:13.432468 30451 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:34227
I20260928 12:35:13.432503 30518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:34227 every 8 connection(s)
I20260928 12:35:13.432849 30451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:13.433877 30519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:13.435487 30519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:35:13.437029 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 30451
I20260928 12:35:13.437143 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:13.437318 8244 external_mini_cluster.cc:1428] Setting key 3717e9953ce946d2475e4949c2cf97d7
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.61@KRBTEST.COM; defaulting to no policy
I20260928 12:35:13.445552 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.436178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34227 (local address 127.0.0.1:43932)
0928 12:35:13.436350 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.436391 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:13.436918 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:13.437246 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:13.437249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:13.437557 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:13.437725 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.437730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.439203 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.439205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.439904 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.439906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.440402 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.440403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.440480 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.440987 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:13.440996 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:13.442416 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:13.444133 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.444140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.444142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.444333 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.444546 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.444546 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.444547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.444625 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.444967 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:13.444971 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:13.445035 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 12:35:13.445133 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 12:35:13.445358 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":76,"thread_start_us":82,"threads_started":1}
I20260928 12:35:13.445552 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.436205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.8.13.62:34227)
0928 12:35:13.436369 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.436379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.436483 (+ 104us) server_negotiation.cc:407] Connection header received
0928 12:35:13.437095 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.437098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.437120 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.437231 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.437845 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.439106 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.439990 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.440306 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.440411 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.442500 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.442507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.442508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.442514 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.444049 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.444405 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.444408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.444409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.444487 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.444679 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.444680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.444680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.444905 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.445018 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.445163 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.445286 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260928 12:35:13.446406 30519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.61@KRBTEST.COM" created.
I20260928 12:35:13.446833 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.446708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:37105 (local address 127.0.0.1:54978)
0928 12:35:13.446752 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.446801 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 12:35:13.446959 30467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.61:37105: connect: Connection refused (error 111)
I20260928 12:35:13.447470 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.447370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:37105 (local address 127.0.0.1:54988)
0928 12:35:13.447415 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.447445 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:35:13.447572 30519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.61:37105: Network error: Client connection negotiation failed: client connection to 127.8.13.61:37105: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab.
Entry for principal kudu/127.8.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab.
I20260928 12:35:13.460945 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.61:37105
--webserver_interface=127.8.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab
--principal=kudu/127.8.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:34227,127.8.13.61:37105,127.8.13.60:44493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:35:13.496831 30519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } attempt: 1
I20260928 12:35:13.497400 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.497247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:37105 (local address 127.0.0.1:55002)
0928 12:35:13.497320 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.497368 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 12:35:13.497741 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.497651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:37105 (local address 127.0.0.1:55016)
0928 12:35:13.497691 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.497721 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 12:35:13.497853 30519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.61:37105: Network error: Client connection negotiation failed: client connection to 127.8.13.61:37105: connect: Connection refused (error 111)
W20260928 12:35:13.540915 30526 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:35:13.541141 30526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:13.541157 30526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:13.541177 30526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:13.542318 30526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:13.542357 30526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:13.542371 30526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:13.542382 30526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:13.542392 30526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:13.543890 30526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:34227,127.8.13.61:37105,127.8.13.60:44493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab
--principal=kudu/127.8.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.61:37105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:13.544134 30526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:13.544335 30526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:13.548945 30531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:13.548946 30532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:13.549028 30526 server_base.cc:1034] running on GCE node
W20260928 12:35:13.548938 30534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:13.549391 30526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:13.549705 30526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:13.550854 30526 hybrid_clock.cc:648] HybridClock initialized: now 1790598913550843 us; error 29 us; skew 500 ppm
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:13.553284 30526 init.cc:377] Logged in from keytab as kudu/127.8.13.61@KRBTEST.COM (short username kudu)
I20260928 12:35:13.553745 30526 webserver.cc:460] Webserver started at http://127.8.13.61:38227/ using document root <none> and password file <none>
I20260928 12:35:13.553995 30526 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:13.554040 30526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:13.554158 30526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:13.555213 30526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/data/instance:
uuid: "cc1b1b5664914cf48552bf3c8e586e2a"
format_stamp: "Formatted at 2026-09-28 12:35:13 on dist-test-slave-g31s"
server_key: "3d57b9b69455794b49fa3714c6c88e66"
server_key_iv: "9cb79f5ea740e00fad416209a7f991b6"
server_key_version: "encryptionkey@0"
I20260928 12:35:13.555528 30526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/wal/instance:
uuid: "cc1b1b5664914cf48552bf3c8e586e2a"
format_stamp: "Formatted at 2026-09-28 12:35:13 on dist-test-slave-g31s"
server_key: "3d57b9b69455794b49fa3714c6c88e66"
server_key_iv: "9cb79f5ea740e00fad416209a7f991b6"
server_key_version: "encryptionkey@0"
I20260928 12:35:13.556806 30526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:35:13.557446 30541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:13.557646 30526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:35:13.557744 30526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/wal
uuid: "cc1b1b5664914cf48552bf3c8e586e2a"
format_stamp: "Formatted at 2026-09-28 12:35:13 on dist-test-slave-g31s"
server_key: "3d57b9b69455794b49fa3714c6c88e66"
server_key_iv: "9cb79f5ea740e00fad416209a7f991b6"
server_key_version: "encryptionkey@0"
I20260928 12:35:13.557855 30526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:13.566037 30519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } attempt: 2
I20260928 12:35:13.566504 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.566382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:37105 (local address 127.0.0.1:55032)
0928 12:35:13.566437 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.566473 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 12:35:13.566790 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.566708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:37105 (local address 127.0.0.1:55046)
0928 12:35:13.566744 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.566768 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 12:35:13.566890 30519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.61:37105: Network error: Client connection negotiation failed: client connection to 127.8.13.61:37105: connect: Connection refused (error 111)
I20260928 12:35:13.576395 30526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:13.578622 30526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:13.578758 30526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:13.583062 30526 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.61:37105
I20260928 12:35:13.583232 30593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.61:37105 every 8 connection(s)
I20260928 12:35:13.583506 30526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
I20260928 12:35:13.584386 30594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:13.585832 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 30526
I20260928 12:35:13.585928 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-1/wal/instance
I20260928 12:35:13.586093 8244 external_mini_cluster.cc:1428] Setting key 177d939cbe7f536163d01d3eece2a44c
I20260928 12:35:13.586171 30594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.60@KRBTEST.COM; defaulting to no policy
I20260928 12:35:13.594847 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.586864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.8.13.62:34227)
0928 12:35:13.586929 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.586931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.587133 (+ 202us) server_negotiation.cc:407] Connection header received
0928 12:35:13.587631 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.587632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.587651 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.587701 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.588321 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.588800 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.589754 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.590002 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.590060 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.592292 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.592293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.592294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.592301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.593610 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.594055 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.594056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.594056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.594107 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.594283 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.594284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.594284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.594423 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.594507 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.594708 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.594777 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:35:13.595008 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.586821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34227 (local address 127.0.0.1:43942)
0928 12:35:13.587008 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.587051 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:13.587537 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:13.587780 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:13.587784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:13.588033 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:13.588237 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.588241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.588881 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.588882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.589673 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.589675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.590100 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.590102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.590222 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.590736 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:13.590744 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:13.592193 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:13.593712 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.593733 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.593735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.593996 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.594152 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.594152 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.594153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.594246 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.594497 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:13.594500 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:13.594614 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 12:35:13.594686 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 12:35:13.594819 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.60@KRBTEST.COM" created.
I20260928 12:35:13.595547 30594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
I20260928 12:35:13.603214 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.595811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:37105 (local address 127.0.0.1:55054)
0928 12:35:13.595857 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.595869 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:13.595908 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:13.596103 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:13.596103 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:13.596242 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:13.596303 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.596304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.597449 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.597450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.598062 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.598065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.598675 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.598676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.598749 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.599194 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:13.599199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:13.600463 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:13.602246 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.602247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.602248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.602443 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.602622 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.602623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.602623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.602688 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.602929 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:13.602930 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:13.602971 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 12:35:13.603024 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:35:13.603111 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 12:35:13.603287 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.595848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.8.13.61:37105)
0928 12:35:13.595982 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.595986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.596004 (+ 18us) server_negotiation.cc:407] Connection header received
0928 12:35:13.596024 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.596026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.596036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.596093 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.596363 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.597360 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.598132 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.598580 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.598656 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.600572 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.600582 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.600582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.600588 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.602164 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.602515 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.602517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.602518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.602565 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.602741 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.602742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.602742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.602875 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.602934 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.603049 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.603155 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260928 12:35:13.603938 30594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:35:13.604312 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.604162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:59942)
0928 12:35:13.604231 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.604280 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 12:35:13.604447 30544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
I20260928 12:35:13.604828 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.604735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:59952)
0928 12:35:13.604783 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.604807 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 12:35:13.604921 30594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:44493: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab.
Entry for principal kudu/127.8.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab.
I20260928 12:35:13.612027 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.60:44493
--webserver_interface=127.8.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab
--principal=kudu/127.8.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:34227,127.8.13.61:37105,127.8.13.60:44493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:35:13.654157 30594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } attempt: 1
I20260928 12:35:13.654657 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.654497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:59966)
0928 12:35:13.654563 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.654614 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 12:35:13.655042 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.654951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:59970)
0928 12:35:13.654994 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.655020 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 12:35:13.655141 30594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:44493: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
I20260928 12:35:13.658030 30519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } attempt: 3
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
I20260928 12:35:13.664918 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.658420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.8.13.61:37105)
0928 12:35:13.658478 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.658481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.658493 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:35:13.658523 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.658524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.658536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.658577 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.658911 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.659425 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.660190 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.660412 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.660467 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.662622 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.662623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.662623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.662630 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.663880 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.664237 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.664238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.664238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.664285 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.664486 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.664486 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.664487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.664591 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.664637 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.664791 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.664854 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:35:13.664965 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.658361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:37105 (local address 127.0.0.1:55070)
0928 12:35:13.658409 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.658421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:13.658472 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:13.658598 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:13.658607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:13.658766 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:13.658837 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.658840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.659509 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.659511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.660107 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.660110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.660471 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.660473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.660550 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.661007 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:13.661012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:13.662538 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:13.663955 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.663957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.663957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.664174 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.664372 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.664373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.664373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.664435 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.664640 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:13.664641 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:13.664697 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 12:35:13.664766 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 12:35:13.664869 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 12:35:13.665416 30519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:35:13.665752 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.665641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:59972)
0928 12:35:13.665685 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.665725 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 12:35:13.666054 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.665973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:59982)
0928 12:35:13.666011 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.666036 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 12:35:13.666188 30519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:44493: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
I20260928 12:35:13.697355 30519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } attempt: 1
W20260928 12:35:13.697244 30601 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:35:13.697518 30601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:13.697553 30601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:13.697593 30601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 12:35:13.697867 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.697685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:59994)
0928 12:35:13.697746 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.697792 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 12:35:13.698169 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.698075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:60000)
0928 12:35:13.698117 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.698147 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:35:13.698275 30519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:44493: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
W20260928 12:35:13.699076 30601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:13.699127 30601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:13.699141 30601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:13.699152 30601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:13.699162 30601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:13.700786 30601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:34227,127.8.13.61:37105,127.8.13.60:44493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab
--principal=kudu/127.8.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.60:44493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:13.701074 30601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:13.701313 30601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:35:13.706050 30601 server_base.cc:1034] running on GCE node
W20260928 12:35:13.706024 30606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:13.706004 30609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:13.706023 30607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:13.706476 30601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:13.706740 30601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:13.707880 30601 hybrid_clock.cc:648] HybridClock initialized: now 1790598913707868 us; error 29 us; skew 500 ppm
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:13.710296 30601 init.cc:377] Logged in from keytab as kudu/127.8.13.60@KRBTEST.COM (short username kudu)
I20260928 12:35:13.710692 30601 webserver.cc:460] Webserver started at http://127.8.13.60:41657/ using document root <none> and password file <none>
I20260928 12:35:13.710898 30601 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:13.710951 30601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:13.711076 30601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:13.712080 30601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/data/instance:
uuid: "134592b9fb8c47f8848ae7c01b9cfd89"
format_stamp: "Formatted at 2026-09-28 12:35:13 on dist-test-slave-g31s"
server_key: "eb97e6e820f15ccc68ab707dc4167dac"
server_key_iv: "8e1b25720ada5146e55f78e6d3d36944"
server_key_version: "encryptionkey@0"
I20260928 12:35:13.712442 30601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/wal/instance:
uuid: "134592b9fb8c47f8848ae7c01b9cfd89"
format_stamp: "Formatted at 2026-09-28 12:35:13 on dist-test-slave-g31s"
server_key: "eb97e6e820f15ccc68ab707dc4167dac"
server_key_iv: "8e1b25720ada5146e55f78e6d3d36944"
server_key_version: "encryptionkey@0"
I20260928 12:35:13.713665 30601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:35:13.714334 30616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:13.714476 30601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:13.714558 30601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/wal
uuid: "134592b9fb8c47f8848ae7c01b9cfd89"
format_stamp: "Formatted at 2026-09-28 12:35:13 on dist-test-slave-g31s"
server_key: "eb97e6e820f15ccc68ab707dc4167dac"
server_key_iv: "8e1b25720ada5146e55f78e6d3d36944"
server_key_version: "encryptionkey@0"
I20260928 12:35:13.714623 30601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:13.723271 30594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } attempt: 2
I20260928 12:35:13.723712 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.723578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:60006)
0928 12:35:13.723640 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.723680 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 12:35:13.724069 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.723975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:60022)
0928 12:35:13.724017 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.724044 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:35:13.724151 30594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:44493: Network error: Client connection negotiation failed: client connection to 127.8.13.60:44493: connect: Connection refused (error 111)
I20260928 12:35:13.730418 30601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:13.732576 30601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:13.732724 30601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:13.736528 30601 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.60:44493
I20260928 12:35:13.736567 30668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.60:44493 every 8 connection(s)
I20260928 12:35:13.736923 30601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
I20260928 12:35:13.737929 30669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:13.739528 30669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:35:13.746937 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 30601
I20260928 12:35:13.747054 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598640018661-8244-0/minicluster-data/master-2/wal/instance
I20260928 12:35:13.747246 8244 external_mini_cluster.cc:1428] Setting key c1bdccc20adb76e642815a57ee3c5786
I20260928 12:35:13.748474 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.740237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.8.13.62:34227)
0928 12:35:13.740296 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.740298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.740485 (+ 187us) server_negotiation.cc:407] Connection header received
0928 12:35:13.740946 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.740947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.740955 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.741015 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.741595 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.742123 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.743114 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.743308 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.743364 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.745557 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.745559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.745560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.745569 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.746795 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.747315 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.747316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.747317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.747386 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.747691 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.747692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.747693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.747851 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.747923 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.748315 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.748400 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:35:13.748708 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.740205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:34227 (local address 127.0.0.1:43950)
0928 12:35:13.740370 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.740412 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:13.740866 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:13.741068 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:13.741070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:13.741325 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:13.741524 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.741528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.742200 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.742201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.743031 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.743033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.743391 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.743392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.743502 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.744044 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:13.744052 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:13.745451 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:13.746882 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.746894 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.746898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.747235 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.747459 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.747460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.747460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.747579 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.747922 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:13.747927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:13.748109 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 12:35:13.748267 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 12:35:13.748460 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260928 12:35:13.749281 30669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
I20260928 12:35:13.755928 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.749588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55078 (local address 127.8.13.61:37105)
0928 12:35:13.749655 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.749657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.749664 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:35:13.749688 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.749689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.749698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.749759 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.750076 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.750520 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.751212 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.751407 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.751473 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.753657 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.753658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.753659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.753666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.754939 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.755227 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.755229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.755229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.755283 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.755451 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.755451 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.755452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.755563 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.755616 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.755751 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.755830 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:35:13.755980 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.747668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.8.13.60:44493)
0928 12:35:13.747851 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.747856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.747885 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:35:13.747911 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.747914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.747930 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.747998 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.748857 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.749824 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.750588 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.750933 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.750987 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.753251 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.753265 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.753265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.753272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.754783 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.755098 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.755102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.755102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.755187 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.755352 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.755353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.755353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.755520 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.755649 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.755733 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.755846 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260928 12:35:13.756163 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.749515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:37105 (local address 127.0.0.1:55078)
0928 12:35:13.749553 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.749562 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:13.749594 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:13.749762 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:13.749763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:13.749923 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:13.749977 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.749979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.750607 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.750608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.751135 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.751138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.751489 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.751490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.751552 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.751944 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:13.751948 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:13.753572 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:13.755012 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.755013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.755014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.755167 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.755336 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.755337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.755337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.755403 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.755614 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:13.755615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:13.755657 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 12:35:13.755716 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:35:13.755914 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":98}
I20260928 12:35:13.756702 30669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:35:13.763407 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.756949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:60034)
0928 12:35:13.756979 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.756987 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:13.757025 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:13.757128 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:13.757129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:13.757278 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:13.757310 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.757311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.757952 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.757952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.758508 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.758510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.758843 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.758844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.758886 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.759257 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:13.759263 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:13.760729 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:13.762420 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.762422 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.762422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.762622 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.762804 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.762804 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.762805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.762870 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.763094 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:13.763095 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:13.763153 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 12:35:13.763212 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:35:13.763308 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 12:35:13.763489 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.756980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60034 (local address 127.8.13.60:44493)
0928 12:35:13.757025 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.757027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.757034 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:35:13.757071 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.757072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.757080 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.757117 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.757369 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.757895 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.758575 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.758789 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.758842 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.760818 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.760820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.760820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.760828 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.762332 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.762693 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.762694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.762694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.762749 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.762930 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.762931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.762931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.763034 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.763082 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.763248 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.763356 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":29}
I20260928 12:35:13.764153 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.756927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.8.13.62:34227)
0928 12:35:13.756978 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.756980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.756987 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:35:13.757012 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.757013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.757022 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.757067 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.757510 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.758020 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.758710 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.758896 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.758949 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.761410 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.761412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.761412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.761420 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.763038 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.763408 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.763409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.763410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.763470 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.763680 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.763681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.763682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.763801 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.763981 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.764013 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.764077 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:35:13.765065 30669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89: Bootstrap starting.
I20260928 12:35:13.765904 30669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:13.766239 30669 log.cc:824] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:13.767022 30669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89: No bootstrap required, opened a new log
I20260928 12:35:13.769114 30669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } }
I20260928 12:35:13.769306 30669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 134592b9fb8c47f8848ae7c01b9cfd89, State: Initialized, Role: FOLLOWER
I20260928 12:35:13.769402 30669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } }
I20260928 12:35:13.769764 30669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:13.769730 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } } }
I20260928 12:35:13.769834 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
W20260928 12:35:13.771574 30686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:35:13.771631 30686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:35:13.773432 30519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } attempt: 2
I20260928 12:35:13.774231 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.765904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.8.13.61:37105)
0928 12:35:13.765969 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.765971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.765982 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:35:13.766009 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.766010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.766022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.766081 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.766633 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.767154 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.768085 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.768301 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.768346 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.770925 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.770927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.770928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.770936 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.772614 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.773022 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.773024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.773024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.773097 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.773299 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.773300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.773300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.773441 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.773598 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.774054 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.774127 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:35:13.772087 30601 master_runner.cc:186] Error getting master registration for 127.8.13.62:34227: OK, instance_id { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" instance_seqno: 1790598913431419 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:35:13.775032 30601 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:35:13.775520 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
I20260928 12:35:13.776966 30493 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a32e17ce622f450f83148bba65ced1f9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43954
I20260928 12:35:13.780540 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.773944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.8.13.60:44493)
0928 12:35:13.773985 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.773986 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.773994 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:35:13.774015 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.774016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.774026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.774067 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.774547 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.775204 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.776131 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.776435 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.776514 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.778293 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.778294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.778294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.778302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.779511 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.779824 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.779825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.779826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.779873 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.780032 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.780033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.780033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.780173 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.780239 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.780372 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.780445 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:35:13.780540 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.773690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:60050)
0928 12:35:13.773782 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.773792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:13.773860 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:13.774132 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:13.774133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:13.774303 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:13.774355 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.774358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.775305 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.775306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.776037 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.776040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.776512 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.776514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.776560 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.776958 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:13.776963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:13.778204 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:13.779590 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.779591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.779591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.779764 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.779927 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.779927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.779927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.779984 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.780234 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:13.780235 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:13.780283 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 12:35:13.780343 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:35:13.780444 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 12:35:13.782459 30519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9: Bootstrap starting.
I20260928 12:35:13.783170 30519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:13.783478 30519 log.cc:824] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:13.784305 30519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9: No bootstrap required, opened a new log
I20260928 12:35:13.785647 30519 raft_consensus.cc:348] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } }
I20260928 12:35:13.785835 30519 raft_consensus.cc:729] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32e17ce622f450f83148bba65ced1f9, State: Initialized, Role: FOLLOWER
I20260928 12:35:13.785928 30519 consensus_queue.cc:260] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } }
I20260928 12:35:13.786226 30689 sys_catalog.cc:455] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } } }
I20260928 12:35:13.786270 30519 sys_catalog.cc:564] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:13.786391 30689 sys_catalog.cc:458] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:13.786283 30493 raft_consensus.cc:482] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 12:35:13.787495 30493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:13.788149 30493 raft_consensus.cc:504] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } }
I20260928 12:35:13.788743 30643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a32e17ce622f450f83148bba65ced1f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "134592b9fb8c47f8848ae7c01b9cfd89"
I20260928 12:35:13.788892 30643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [term 0 FOLLOWER]: Advancing to term 1
W20260928 12:35:13.789216 30700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:35:13.789275 30700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:35:13.789856 30643 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32e17ce622f450f83148bba65ced1f9 in term 1.
I20260928 12:35:13.789872 30493 leader_election.cc:288] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [CANDIDATE]: Term 1 election: Requested vote from peers cc1b1b5664914cf48552bf3c8e586e2a (127.8.13.61:37105), 134592b9fb8c47f8848ae7c01b9cfd89 (127.8.13.60:44493)
I20260928 12:35:13.790407 30469 leader_election.cc:302] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134592b9fb8c47f8848ae7c01b9cfd89, a32e17ce622f450f83148bba65ced1f9; no voters:
I20260928 12:35:13.790555 30689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:13.790696 30689 raft_consensus.cc:686] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [term 1 LEADER]: Becoming Leader. State: Replica: a32e17ce622f450f83148bba65ced1f9, State: Running, Role: LEADER
I20260928 12:35:13.790792 30689 consensus_queue.cc:237] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } }
I20260928 12:35:13.791221 30689 sys_catalog.cc:455] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a32e17ce622f450f83148bba65ced1f9. Latest consensus state: current_term: 1 leader_uuid: "a32e17ce622f450f83148bba65ced1f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } } }
I20260928 12:35:13.791311 30689 sys_catalog.cc:458] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:13.790045 30451 master_runner.cc:186] Error getting master registration for 127.8.13.61:37105: OK, instance_id { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" instance_seqno: 1790598913581765 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:35:13.792035 30451 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:35:13.794005 30708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:13.794216 30708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:13.795698 30643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [term 1 FOLLOWER]: Refusing update from remote peer a32e17ce622f450f83148bba65ced1f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:35:13.796051 30705 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:35:13.797400 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a32e17ce622f450f83148bba65ced1f9. Latest consensus state: current_term: 1 leader_uuid: "a32e17ce622f450f83148bba65ced1f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } } }
I20260928 12:35:13.797497 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:13.798941 30705 sys_catalog.cc:455] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a32e17ce622f450f83148bba65ced1f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } } }
I20260928 12:35:13.799029 30705 sys_catalog.cc:458] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:13.799409 30709 mvcc.cc:205] Tried to move back new op lower bound from 7334293150904401920 to 7334293150887804928. Current Snapshot: MvccSnapshot[applied={T|T < 7334293150904401920}]
I20260928 12:35:13.799628 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a32e17ce622f450f83148bba65ced1f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } } }
I20260928 12:35:13.799713 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:13.801476 30713 mvcc.cc:205] Tried to move back new op lower bound from 7334293150904401920 to 7334293150887804928. Current Snapshot: MvccSnapshot[applied={T|T < 7334293150904401920}]
I20260928 12:35:13.806170 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.795081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.8.13.60:44493)
0928 12:35:13.795131 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.795133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.797595 (+ 2462us) server_negotiation.cc:407] Connection header received
0928 12:35:13.797694 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.797695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.797705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.797735 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.798583 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.799312 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.800692 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.801001 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.801071 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.802704 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.802705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.802706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.802713 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.804248 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.805138 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.805139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.805140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.805202 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.805403 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.805404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.805404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.805525 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.805630 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.805993 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.806069 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:35:13.806725 30708 catalog_manager.cc:1347] Generated new cluster ID: 09f042eecd91435ab63481219ba8e2ab
I20260928 12:35:13.806782 30708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:35:13.810264 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.795551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.8.13.61:37105)
0928 12:35:13.795590 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.795591 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.802112 (+ 6521us) server_negotiation.cc:407] Connection header received
0928 12:35:13.802704 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.802706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.802729 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.802759 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.803296 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.803974 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.805524 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.806282 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.806326 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.807453 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.807454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.807454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.807473 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.808972 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.809380 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.809381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.809382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.809441 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.809631 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.809631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.809632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.809752 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.809871 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.810081 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.810158 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:35:13.811487 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.795260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.8.13.62:34227)
0928 12:35:13.795297 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.795299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.801931 (+ 6632us) server_negotiation.cc:407] Connection header received
0928 12:35:13.802042 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.802043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.802055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.802120 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.802632 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.803975 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.804726 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.805005 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.805083 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.807759 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.807761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.807761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.807781 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.809577 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.810137 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.810138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.810139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.810200 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.810835 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.810836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.810836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.810968 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.811036 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.811324 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.811396 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:35:13.813378 30708 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:13.814056 30708 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:13.815281 30594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } attempt: 3
Sep 28 12:35:13 dist-test-slave-g31s krb5kdc[30438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598913, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
I20260928 12:35:13.822798 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.815509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:44493 (local address 127.0.0.1:60076)
0928 12:35:13.815560 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:13.815573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:13.815616 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:13.815740 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:13.815740 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:13.815895 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:13.815936 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.815937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.816841 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.816842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.817639 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:13.817641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.818160 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.818161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:13.818229 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.818728 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:13.818733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:13.820153 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:13.821911 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.821912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.821912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.822096 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.822253 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:13.822254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:13.822254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:13.822299 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:13.822517 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:13.822518 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:13.822555 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 12:35:13.822605 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:35:13.822701 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 12:35:13.823216 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.815597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.8.13.60:44493)
0928 12:35:13.815634 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.815636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.815645 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:35:13.815665 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.815666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.815675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.815724 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.815993 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.816751 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.817716 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.818090 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.818150 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.820230 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:13.820231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:13.820231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:13.820238 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:13.821833 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.822157 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.822158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.822159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.822203 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:13.822351 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:13.822351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:13.822351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:13.822457 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:13.822499 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.823064 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.823132 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:35:13.825125 30594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a: Bootstrap starting.
I20260928 12:35:13.826785 30594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:13.827095 30594 log.cc:824] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:13.827369 30708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9: Generated new TSK 0
I20260928 12:35:13.827637 30708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:35:13.828114 30594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a: No bootstrap required, opened a new log
I20260928 12:35:13.830288 30594 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } }
I20260928 12:35:13.830505 30594 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1b1b5664914cf48552bf3c8e586e2a, State: Initialized, Role: FOLLOWER
I20260928 12:35:13.830643 30594 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } }
I20260928 12:35:13.831028 30594 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:13.831099 30567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:13.831976 30567 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [term 1 FOLLOWER]: Refusing update from remote peer a32e17ce622f450f83148bba65ced1f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:35:13.832173 30568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a32e17ce622f450f83148bba65ced1f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cc1b1b5664914cf48552bf3c8e586e2a"
I20260928 12:35:13.833263 30568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32e17ce622f450f83148bba65ced1f9 in term 1.
I20260928 12:35:13.833611 30689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:35:13.833855 30719 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } } }
I20260928 12:35:13.834020 30719 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:13.841462 30730 catalog_manager.cc:1259] Loaded cluster ID: 09f042eecd91435ab63481219ba8e2ab
I20260928 12:35:13.841703 30730 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a: loading cluster ID for follower catalog manager: success
I20260928 12:35:13.843029 30730 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a: acquiring CA information for follower catalog manager: success
I20260928 12:35:13.843518 30730 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:35:13.841614 30719 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a32e17ce622f450f83148bba65ced1f9. Latest consensus state: current_term: 1 leader_uuid: "a32e17ce622f450f83148bba65ced1f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } } }
I20260928 12:35:13.843779 30719 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:13.843864 30719 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a32e17ce622f450f83148bba65ced1f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } } }
I20260928 12:35:13.843969 30719 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc1b1b5664914cf48552bf3c8e586e2a [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:35:13.866066 30673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:13.864441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.8.13.60:44493)
0928 12:35:13.864501 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.864503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.865324 (+ 821us) server_negotiation.cc:407] Connection header received
0928 12:35:13.865429 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.865430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.865442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.865473 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.865987 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60084: BlockingRecv error: recv got EOF from 127.0.0.1:60084 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 12:35:13.866868 30522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:13.864878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.8.13.62:34227)
0928 12:35:13.864926 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.864928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.866324 (+ 1396us) server_negotiation.cc:407] Connection header received
0928 12:35:13.866425 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.866426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.866437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.866466 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.866801 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43970: BlockingRecv error: recv got EOF from 127.0.0.1:43970 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 12:35:13.867543 30598 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:13.865102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.8.13.61:37105)
0928 12:35:13.865140 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.865141 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.867049 (+ 1908us) server_negotiation.cc:407] Connection header received
0928 12:35:13.867114 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.867115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.867125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.867175 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.867479 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55102: BlockingRecv error: recv got EOF from 127.0.0.1:55102 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:35:13.874641 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.870949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.8.13.61:37105)
0928 12:35:13.870988 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.871007 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.871251 (+ 244us) server_negotiation.cc:407] Connection header received
0928 12:35:13.871363 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.871364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.871412 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.871455 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:13.871657 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.872449 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.873611 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.873962 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.874004 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.874230 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:13.874367 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:13.874417 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.874528 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.874551 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:35:13.878193 30673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:13.877276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.8.13.60:44493)
0928 12:35:13.877320 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.877322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.877557 (+ 235us) server_negotiation.cc:407] Connection header received
0928 12:35:13.877664 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.877665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.877685 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.877713 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.878126 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60092: BlockingRecv error: recv got EOF from 127.0.0.1:60092 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:35:13.883937 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.880553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.8.13.61:37105)
0928 12:35:13.880595 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.880597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.880857 (+ 260us) server_negotiation.cc:407] Connection header received
0928 12:35:13.880957 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.880958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.880969 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.880997 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:13.881170 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.881951 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.883117 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.883416 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.883457 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.883647 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:13.883684 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:13.883709 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.883822 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.883848 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 12:35:13.887550 30673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:13.886643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.8.13.60:44493)
0928 12:35:13.886687 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.886688 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.886951 (+ 263us) server_negotiation.cc:407] Connection header received
0928 12:35:13.887048 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.887049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.887061 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.887091 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.887494 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60106: BlockingRecv error: recv got EOF from 127.0.0.1:60106 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:35:13.894193 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.890827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.8.13.61:37105)
0928 12:35:13.890880 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.890882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.891114 (+ 232us) server_negotiation.cc:407] Connection header received
0928 12:35:13.891208 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.891209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.891219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.891248 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:13.891421 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.892191 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.893335 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.893636 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.893676 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.893921 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:13.893960 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:13.893985 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.894080 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.894104 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 12:35:13.897500 30673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:13.896644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.8.13.60:44493)
0928 12:35:13.896688 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.896690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.896950 (+ 260us) server_negotiation.cc:407] Connection header received
0928 12:35:13.897044 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.897045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.897057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.897085 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.897434 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60122: BlockingRecv error: recv got EOF from 127.0.0.1:60122 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:35:13.906131 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.902853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.8.13.61:37105)
0928 12:35:13.902896 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.902897 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.903126 (+ 229us) server_negotiation.cc:407] Connection header received
0928 12:35:13.903220 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.903222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.903234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.903259 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:13.903417 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.904169 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.905301 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.905599 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.905636 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.905868 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:13.905905 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:13.905930 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.906026 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.906046 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 12:35:13.909287 30673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:13.908401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.8.13.60:44493)
0928 12:35:13.908444 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.908446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.908666 (+ 220us) server_negotiation.cc:407] Connection header received
0928 12:35:13.908773 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.908774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.908785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.908812 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.909231 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60130: BlockingRecv error: recv got EOF from 127.0.0.1:60130 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:35:13.921854 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.918599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.8.13.61:37105)
0928 12:35:13.918642 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.918644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.918891 (+ 247us) server_negotiation.cc:407] Connection header received
0928 12:35:13.918988 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.918989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.918999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.919026 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:13.919202 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.919910 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.920997 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.921275 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.921318 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.921520 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:13.921555 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:13.921581 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.921683 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.921704 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:35:13.924937 30673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:13.924139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.8.13.60:44493)
0928 12:35:13.924178 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.924180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.924403 (+ 223us) server_negotiation.cc:407] Connection header received
0928 12:35:13.924494 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.924495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.924505 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.924532 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.924875 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60136: BlockingRecv error: recv got EOF from 127.0.0.1:60136 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:35:13.945828 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.942417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.8.13.61:37105)
0928 12:35:13.942463 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.942465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.942742 (+ 277us) server_negotiation.cc:407] Connection header received
0928 12:35:13.942851 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.942852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.942863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.942891 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:13.943079 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.943837 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.944983 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.945268 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.945314 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.945511 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:13.945547 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:13.945573 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.945673 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.945696 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 12:35:13.949298 30673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:13.948371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.8.13.60:44493)
0928 12:35:13.948416 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.948418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.948669 (+ 251us) server_negotiation.cc:407] Connection header received
0928 12:35:13.948784 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.948785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.948798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.948827 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.949234 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60150: BlockingRecv error: recv got EOF from 127.0.0.1:60150 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:35:13.986121 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:13.982815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.8.13.61:37105)
0928 12:35:13.982867 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.982869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.982970 (+ 101us) server_negotiation.cc:407] Connection header received
0928 12:35:13.983030 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.983031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.983043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.983134 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:13.983300 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.984039 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.985244 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:13.985534 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:13.985664 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:13.985853 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:13.985892 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:13.985940 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:13.986009 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 12:35:13.986031 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 12:35:13.990535 30673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:13.988107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.8.13.60:44493)
0928 12:35:13.988198 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:13.988201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:13.989990 (+ 1789us) server_negotiation.cc:407] Connection header received
0928 12:35:13.990052 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:13.990053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:13.990069 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:13.990131 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:13.990465 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60152: BlockingRecv error: recv got EOF from 127.0.0.1:60152 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 12:35:14.057967 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:14.055790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.8.13.61:37105)
0928 12:35:14.055878 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:14.055880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:14.055945 (+ 65us) server_negotiation.cc:407] Connection header received
0928 12:35:14.055975 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:14.055976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:14.055985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:14.056035 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:14.056139 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.056621 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.057352 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.057556 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.057605 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:14.057725 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:14.057752 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:14.057786 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:14.057872 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 12:35:14.057888 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 12:35:14.060176 30673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:14.059661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.8.13.60:44493)
0928 12:35:14.059719 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:14.059721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:14.059773 (+ 52us) server_negotiation.cc:407] Connection header received
0928 12:35:14.059828 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:14.059829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:14.059838 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:14.059880 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:14.060126 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60160: BlockingRecv error: recv got EOF from 127.0.0.1:60160 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:35:14.191490 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:14.189265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.8.13.61:37105)
0928 12:35:14.189321 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:14.189323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:14.189402 (+ 79us) server_negotiation.cc:407] Connection header received
0928 12:35:14.189451 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:14.189452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:14.189462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:14.189522 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:14.189676 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.190224 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.190951 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.191151 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.191199 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:14.191316 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:14.191343 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:14.191372 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:14.191410 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 12:35:14.191425 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 12:35:14.193738 30673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:14.193112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.8.13.60:44493)
0928 12:35:14.193201 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:14.193203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:14.193252 (+ 49us) server_negotiation.cc:407] Connection header received
0928 12:35:14.193326 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:14.193327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:14.193339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:14.193382 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:14.193681 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60172: BlockingRecv error: recv got EOF from 127.0.0.1:60172 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 12:35:14.411659 30837 sys_catalog.cc:455] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a32e17ce622f450f83148bba65ced1f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32e17ce622f450f83148bba65ced1f9" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 34227 } } peers { permanent_uuid: "cc1b1b5664914cf48552bf3c8e586e2a" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 37105 } } peers { permanent_uuid: "134592b9fb8c47f8848ae7c01b9cfd89" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 44493 } } }
I20260928 12:35:14.411788 30837 sys_catalog.cc:458] T 00000000000000000000000000000000 P a32e17ce622f450f83148bba65ced1f9 [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:14.452888 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:14.450838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.8.13.61:37105)
0928 12:35:14.450907 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:14.450909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:14.450922 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:35:14.450989 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:14.450990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:14.451001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:14.451050 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:14.451161 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.451678 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.452376 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.452553 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.452597 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:14.452693 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:14.452719 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:14.452750 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:14.452789 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 12:35:14.452803 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 12:35:14.454949 30673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:14.454471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.8.13.60:44493)
0928 12:35:14.454533 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:14.454535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:14.454543 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:35:14.454569 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:14.454570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:14.454579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:14.454626 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:14.454896 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60178: BlockingRecv error: recv got EOF from 127.0.0.1:60178 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:35:14.772060 30686 catalog_manager.cc:1259] Loaded cluster ID: 09f042eecd91435ab63481219ba8e2ab
I20260928 12:35:14.772138 30686 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89: loading cluster ID for follower catalog manager: success
I20260928 12:35:14.772902 30686 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89: acquiring CA information for follower catalog manager: success
I20260928 12:35:14.773160 30686 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 134592b9fb8c47f8848ae7c01b9cfd89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:35:14.970501 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:14.968056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.8.13.61:37105)
0928 12:35:14.968156 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:14.968158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:14.968190 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:35:14.968285 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:14.968286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:14.968296 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:14.968354 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:14.968455 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.968950 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.969758 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.970034 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.970095 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:14.970225 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:14.970261 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:14.970302 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:14.970381 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 12:35:14.970405 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
I20260928 12:35:14.974468 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:14.972088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.8.13.60:44493)
0928 12:35:14.972193 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:14.972195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:14.972203 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:35:14.972229 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:14.972230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:14.972299 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:14.972365 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:14.972469 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.972961 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.973663 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.973936 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.973989 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:14.974135 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:14.974253 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:14.974320 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:14.974355 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 12:35:14.974379 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":52,"threads_started":1}
I20260928 12:35:14.978832 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:14.976190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.8.13.61:37105)
0928 12:35:14.976234 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:14.976236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:14.976263 (+ 27us) server_negotiation.cc:407] Connection header received
0928 12:35:14.976332 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:14.976333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:14.976344 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:14.976387 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:14.976548 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.977024 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.978020 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.978335 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.978405 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:14.978573 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:14.978621 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:14.978666 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:14.978719 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 12:35:14.978746 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:35:14.979110 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:14.975917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.8.13.62:34227)
0928 12:35:14.976022 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:14.976024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:14.976076 (+ 52us) server_negotiation.cc:407] Connection header received
0928 12:35:14.976215 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:14.976216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:14.976279 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:14.976358 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:14.976464 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.977263 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.978210 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.978438 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.978498 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:14.978682 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:14.978803 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:14.978854 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:14.979006 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 12:35:14.979033 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":48,"threads_started":1}
I20260928 12:35:14.979539 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:14.975923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.8.13.60:44493)
0928 12:35:14.975993 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:14.975995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:14.976196 (+ 201us) server_negotiation.cc:407] Connection header received
0928 12:35:14.976294 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:14.976295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:14.976305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:14.976331 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:14.976528 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.977277 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.978811 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:14.979096 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:14.979135 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:14.979308 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:14.979346 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:14.979395 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:14.979450 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 12:35:14.979473 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 12:35:14.984057 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 30451
I20260928 12:35:14.989197 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 30526
I20260928 12:35:14.993918 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 30601
2026-09-28T12:35:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2753 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:35:15 dist-test-slave-g31s krb5kdc[30872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:35:15 dist-test-slave-g31s krb5kdc[30872](info): set up 2 sockets
Sep 28 12:35:15 dist-test-slave-g31s krb5kdc[30872](info): commencing operation
krb5kdc: starting...
W20260928 12:35:15.937961 8244 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/test-user.keytab.
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:35:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
Entry for principal kudu/127.8.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab.
I20260928 12:35:16.054472 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:44853
--webserver_interface=127.8.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:44853,127.8.13.61:46165,127.8.13.60:34037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:35:16.133755 30885 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:35:16.134006 30885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:16.134021 30885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:16.134042 30885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:16.135159 30885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:16.135203 30885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:16.135216 30885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:16.135228 30885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:16.135238 30885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:16.136703 30885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:44853,127.8.13.61:46165,127.8.13.60:34037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.62.keytab
--principal=kudu/127.8.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.62:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:16.136945 30885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:16.137131 30885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:16.142256 30890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:16.142249 30893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:16.142256 30891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:16.142287 30885 server_base.cc:1034] running on GCE node
I20260928 12:35:16.142803 30885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:16.143113 30885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:16.144250 30885 hybrid_clock.cc:648] HybridClock initialized: now 1790598916144201 us; error 70 us; skew 500 ppm
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:16.146739 30885 init.cc:377] Logged in from keytab as kudu/127.8.13.62@KRBTEST.COM (short username kudu)
I20260928 12:35:16.147202 30885 webserver.cc:460] Webserver started at http://127.8.13.62:37323/ using document root <none> and password file <none>
I20260928 12:35:16.147413 30885 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:16.147461 30885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:16.147594 30885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:16.148663 30885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/data/instance:
uuid: "ded87bc0301c4cd882e83bf7fdcff1c3"
format_stamp: "Formatted at 2026-09-28 12:35:16 on dist-test-slave-g31s"
server_key: "d4671b32421ce5455b84de12a3bdcdbe"
server_key_iv: "bebe8ec1f00ac6c91d587e36ec3c3912"
server_key_version: "encryptionkey@0"
I20260928 12:35:16.149005 30885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/wal/instance:
uuid: "ded87bc0301c4cd882e83bf7fdcff1c3"
format_stamp: "Formatted at 2026-09-28 12:35:16 on dist-test-slave-g31s"
server_key: "d4671b32421ce5455b84de12a3bdcdbe"
server_key_iv: "bebe8ec1f00ac6c91d587e36ec3c3912"
server_key_version: "encryptionkey@0"
I20260928 12:35:16.150338 30885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:35:16.151077 30900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:16.151237 30885 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:35:16.151325 30885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/wal
uuid: "ded87bc0301c4cd882e83bf7fdcff1c3"
format_stamp: "Formatted at 2026-09-28 12:35:16 on dist-test-slave-g31s"
server_key: "d4671b32421ce5455b84de12a3bdcdbe"
server_key_iv: "bebe8ec1f00ac6c91d587e36ec3c3912"
server_key_version: "encryptionkey@0"
I20260928 12:35:16.151388 30885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:16.163558 30885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:16.165663 30885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:16.165844 30885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:16.169763 30885 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.62:44853
I20260928 12:35:16.169826 30952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.62:44853 every 8 connection(s)
I20260928 12:35:16.170198 30885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/data/info.pb
I20260928 12:35:16.171154 30953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:16.172745 30953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:35:16.179257 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 30885
I20260928 12:35:16.179394 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-0/wal/instance
I20260928 12:35:16.179622 8244 external_mini_cluster.cc:1428] Setting key fe4d31186836cf6f71aef4388997e794
I20260928 12:35:16.183287 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.173512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:44853 (local address 127.0.0.1:60582)
0928 12:35:16.173694 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.173746 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:16.174289 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:16.174607 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:16.174609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:16.174886 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:16.175051 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.175055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.176038 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.176039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.176730 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.176732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.177208 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.177209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.177272 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.177833 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:16.177844 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:16.179467 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:16.181505 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.181515 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.181518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.181857 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.182145 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.182145 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.182146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.182261 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.182677 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:16.182684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:16.182765 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 12:35:16.182861 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 12:35:16.183102 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":72,"thread_start_us":96,"threads_started":1}
I20260928 12:35:16.183296 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.173602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.8.13.62:44853)
0928 12:35:16.173753 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.173757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.174432 (+ 675us) server_negotiation.cc:407] Connection header received
0928 12:35:16.174500 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.174501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.174517 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.174594 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.175143 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.175949 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.176809 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.177113 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.177219 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.179567 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.179574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.179574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.179581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.181415 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.181958 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.181961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.181962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.182075 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.182327 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.182328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.182328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.182608 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.182690 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.182899 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.183027 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260928 12:35:16.184687 30953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:35:16.185129 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.184980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:46165 (local address 127.0.0.1:36368)
0928 12:35:16.185029 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.185098 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 12:35:16.186071 30901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.61:46165: connect: Connection refused (error 111)
I20260928 12:35:16.186599 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.186505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:46165 (local address 127.0.0.1:36376)
0928 12:35:16.186548 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.186571 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:35:16.186766 30953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.61:46165: Network error: Client connection negotiation failed: client connection to 127.8.13.61:46165: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.8.13.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab.
Entry for principal kudu/127.8.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab.
I20260928 12:35:16.206295 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.61:46165
--webserver_interface=127.8.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab
--principal=kudu/127.8.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:44853,127.8.13.61:46165,127.8.13.60:34037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:35:16.236025 30953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } attempt: 1
I20260928 12:35:16.236585 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.236413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:46165 (local address 127.0.0.1:36378)
0928 12:35:16.236487 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.236542 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 12:35:16.236969 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.236881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:46165 (local address 127.0.0.1:36384)
0928 12:35:16.236917 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.236947 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 12:35:16.237074 30953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.61:46165: Network error: Client connection negotiation failed: client connection to 127.8.13.61:46165: connect: Connection refused (error 111)
W20260928 12:35:16.286514 30960 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:35:16.286755 30960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:16.286772 30960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:16.286794 30960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:16.288021 30960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:16.288080 30960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:16.288093 30960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:16.288105 30960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:16.288115 30960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:16.289754 30960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:44853,127.8.13.61:46165,127.8.13.60:34037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.61.keytab
--principal=kudu/127.8.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.61:46165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:16.290060 30960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:16.290301 30960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:16.295324 30968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:16.295310 30966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:16.295367 30960 server_base.cc:1034] running on GCE node
W20260928 12:35:16.295295 30965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:16.295696 30960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:16.296017 30960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:16.297165 30960 hybrid_clock.cc:648] HybridClock initialized: now 1790598916297147 us; error 35 us; skew 500 ppm
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:16.299818 30960 init.cc:377] Logged in from keytab as kudu/127.8.13.61@KRBTEST.COM (short username kudu)
I20260928 12:35:16.300304 30960 webserver.cc:460] Webserver started at http://127.8.13.61:38501/ using document root <none> and password file <none>
I20260928 12:35:16.300523 30960 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:16.300570 30960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:16.300694 30960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:16.301752 30960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/data/instance:
uuid: "a8eb56e2aca641b4b7431f47fd417ef2"
format_stamp: "Formatted at 2026-09-28 12:35:16 on dist-test-slave-g31s"
server_key: "488426c44efb4202ee70ea4321db3f53"
server_key_iv: "676c705e891ffa261fc3dd18c365fd77"
server_key_version: "encryptionkey@0"
I20260928 12:35:16.302142 30960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/wal/instance:
uuid: "a8eb56e2aca641b4b7431f47fd417ef2"
format_stamp: "Formatted at 2026-09-28 12:35:16 on dist-test-slave-g31s"
server_key: "488426c44efb4202ee70ea4321db3f53"
server_key_iv: "676c705e891ffa261fc3dd18c365fd77"
server_key_version: "encryptionkey@0"
I20260928 12:35:16.303426 30960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:35:16.304155 30975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:16.304309 30960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:35:16.304388 30960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/wal
uuid: "a8eb56e2aca641b4b7431f47fd417ef2"
format_stamp: "Formatted at 2026-09-28 12:35:16 on dist-test-slave-g31s"
server_key: "488426c44efb4202ee70ea4321db3f53"
server_key_iv: "676c705e891ffa261fc3dd18c365fd77"
server_key_version: "encryptionkey@0"
I20260928 12:35:16.304457 30960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:16.305270 30953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } attempt: 2
I20260928 12:35:16.305681 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.305545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:46165 (local address 127.0.0.1:36392)
0928 12:35:16.305601 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.305644 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:35:16.305995 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.305919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:46165 (local address 127.0.0.1:36404)
0928 12:35:16.305952 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.305977 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.61:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 12:35:16.306079 30953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.61:46165: Network error: Client connection negotiation failed: client connection to 127.8.13.61:46165: connect: Connection refused (error 111)
I20260928 12:35:16.319603 30960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:16.321928 30960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:16.322049 30960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:16.325667 30960 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.61:46165
I20260928 12:35:16.325708 31027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.61:46165 every 8 connection(s)
I20260928 12:35:16.326043 30960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/data/info.pb
I20260928 12:35:16.326972 31028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:16.328446 31028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:35:16.331148 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 30960
I20260928 12:35:16.331252 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-1/wal/instance
I20260928 12:35:16.331456 8244 external_mini_cluster.cc:1428] Setting key 62ae0cee64d16828c45ac0690bf11579
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:35:16.337244 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.329068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.8.13.62:44853)
0928 12:35:16.329120 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.329122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.329329 (+ 207us) server_negotiation.cc:407] Connection header received
0928 12:35:16.329757 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.329758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.329768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.329866 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.330424 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.330884 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.332219 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.332440 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.332496 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.334630 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.334631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.334632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.334639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.335970 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.336352 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.336353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.336354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.336407 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.336626 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.336627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.336627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.336743 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.336798 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.337063 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.337150 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:35:16.337422 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.329037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:44853 (local address 127.0.0.1:60592)
0928 12:35:16.329194 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.329235 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:16.329689 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:16.329923 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:16.329926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:16.330157 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:16.330352 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.330356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.330944 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.330945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.332133 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.332136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.332526 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.332527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.332688 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.333168 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:16.333176 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:16.334538 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:16.336058 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.336067 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.336069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.336286 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.336459 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.336460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.336460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.336571 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.336803 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:16.336810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:16.336948 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 12:35:16.337037 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:35:16.337198 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260928 12:35:16.337980 31028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.13.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.60@KRBTEST.COM" created.
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
I20260928 12:35:16.345690 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.338227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:46165 (local address 127.0.0.1:36410)
0928 12:35:16.338424 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.338434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:16.338474 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:16.338617 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:16.338617 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:16.338729 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:16.338784 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.338812 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.339758 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.339759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.340419 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.340422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.341051 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.341052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.341116 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.341592 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:16.341595 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:16.342891 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:16.344565 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.344567 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.344567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.344779 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.344975 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.344976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.344976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.345043 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.345317 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:16.345318 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:16.345372 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 12:35:16.345445 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 12:35:16.345566 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260928 12:35:16.345716 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.338286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.8.13.61:46165)
0928 12:35:16.338462 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.338467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.338495 (+ 28us) server_negotiation.cc:407] Connection header received
0928 12:35:16.338518 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.338520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.338534 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.338609 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.338870 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.339613 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.340525 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.340968 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.341044 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.342984 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.342995 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.342995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.343002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.344484 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.344860 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.344863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.344863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.344917 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.345101 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.345101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.345101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.345256 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.345316 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.345484 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.345628 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260928 12:35:16.346526 31028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:35:16.346900 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.346760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42036)
0928 12:35:16.346808 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.346869 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 12:35:16.347054 30979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
I20260928 12:35:16.347506 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.347403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42042)
0928 12:35:16.347450 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.347480 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:35:16.347610 31028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:34037: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab.
Entry for principal kudu/127.8.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab.
I20260928 12:35:16.356199 8244 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2Ofqh2/build/release/bin/kudu
/tmp/dist-test-task2Ofqh2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.60:34037
--webserver_interface=127.8.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab
--principal=kudu/127.8.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.62:44853,127.8.13.61:46165,127.8.13.60:34037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:35:16.396879 31028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } attempt: 1
I20260928 12:35:16.397221 30953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } attempt: 3
I20260928 12:35:16.397481 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.397323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42046)
0928 12:35:16.397401 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.397439 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 12:35:16.397905 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.397820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42052)
0928 12:35:16.397854 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.397878 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:35:16.397982 31028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:34037: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
I20260928 12:35:16.403571 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.397521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.8.13.61:46165)
0928 12:35:16.397573 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.397575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.397611 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:35:16.397701 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.397702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.397713 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.397763 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.398147 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.398637 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.399308 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.399516 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.399572 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.401367 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.401369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.401369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.401376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.402624 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.402963 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.402963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.402964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.403008 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.403165 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.403166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.403166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.403273 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.403316 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.403456 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.403512 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:35:16.403636 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.397498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:46165 (local address 127.0.0.1:36420)
0928 12:35:16.397554 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.397566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:16.397621 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:16.397770 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:16.397771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:16.397981 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:16.398061 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.398064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.398725 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.398726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.399227 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.399228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.399579 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.399580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.399652 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.400090 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:16.400095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:16.401286 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:16.402692 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.402693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.402694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.402900 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.403056 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.403057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.403057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.403117 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.403320 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:16.403321 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:16.403372 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 12:35:16.403440 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:35:16.403533 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:35:16.404206 30953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:35:16.404531 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.404437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42056)
0928 12:35:16.404479 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.404510 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 12:35:16.404786 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.404710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42072)
0928 12:35:16.404747 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.404770 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 12:35:16.404861 30953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:34037: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
I20260928 12:35:16.436029 30953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } attempt: 1
I20260928 12:35:16.436547 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.436393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42080)
0928 12:35:16.436458 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.436508 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 12:35:16.436889 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.436795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42096)
0928 12:35:16.436837 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.436867 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 12:35:16.436987 30953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:34037: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
W20260928 12:35:16.438973 31035 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:35:16.439201 31035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:35:16.439219 31035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:35:16.439240 31035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:35:16.440416 31035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:35:16.440454 31035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:35:16.440465 31035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:35:16.440476 31035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:35:16.440495 31035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:35:16.442086 31035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.62:44853,127.8.13.61:46165,127.8.13.60:34037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/krb5kdc/kudu_127.8.13.60.keytab
--principal=kudu/127.8.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.60:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:35:16.442327 31035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:35:16.442517 31035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:35:16.447402 31040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:35:16.447402 31041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:16.447584 31035 server_base.cc:1034] running on GCE node
W20260928 12:35:16.447415 31043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:35:16.447932 31035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:35:16.448222 31035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:35:16.449378 31035 hybrid_clock.cc:648] HybridClock initialized: now 1790598916449365 us; error 30 us; skew 500 ppm
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:35:16.451876 31035 init.cc:377] Logged in from keytab as kudu/127.8.13.60@KRBTEST.COM (short username kudu)
I20260928 12:35:16.452384 31035 webserver.cc:460] Webserver started at http://127.8.13.60:41925/ using document root <none> and password file <none>
I20260928 12:35:16.452586 31035 fs_manager.cc:359] Metadata directory not provided
I20260928 12:35:16.452637 31035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:35:16.452762 31035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:35:16.453873 31035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/data/instance:
uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f"
format_stamp: "Formatted at 2026-09-28 12:35:16 on dist-test-slave-g31s"
server_key: "80ca883a3190811b5c8171d481313e40"
server_key_iv: "e3d2daba89488b5bee1331118f3a98f1"
server_key_version: "encryptionkey@0"
I20260928 12:35:16.454232 31035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/wal/instance:
uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f"
format_stamp: "Formatted at 2026-09-28 12:35:16 on dist-test-slave-g31s"
server_key: "80ca883a3190811b5c8171d481313e40"
server_key_iv: "e3d2daba89488b5bee1331118f3a98f1"
server_key_version: "encryptionkey@0"
I20260928 12:35:16.455546 31035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:35:16.456270 31050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:35:16.456449 31035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:35:16.456532 31035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/data,/tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/wal
uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f"
format_stamp: "Formatted at 2026-09-28 12:35:16 on dist-test-slave-g31s"
server_key: "80ca883a3190811b5c8171d481313e40"
server_key_iv: "e3d2daba89488b5bee1331118f3a98f1"
server_key_version: "encryptionkey@0"
I20260928 12:35:16.456604 31035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:35:16.466106 31028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } attempt: 2
I20260928 12:35:16.466557 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.466443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42100)
0928 12:35:16.466497 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.466528 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:35:16.466920 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.466836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42110)
0928 12:35:16.466877 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.466901 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 12:35:16.466993 31028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.60:34037: Network error: Client connection negotiation failed: client connection to 127.8.13.60:34037: connect: Connection refused (error 111)
I20260928 12:35:16.473692 31035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:35:16.475939 31035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:35:16.476086 31035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:35:16.479959 31035 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.60:34037
I20260928 12:35:16.480001 31102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.60:34037 every 8 connection(s)
I20260928 12:35:16.480352 31035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/data/info.pb
I20260928 12:35:16.480916 8244 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2Ofqh2/build/release/bin/kudu as pid 31035
I20260928 12:35:16.481014 8244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2Ofqh2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598640018661-8244-0/minicluster-data/master-2/wal/instance
I20260928 12:35:16.481208 8244 external_mini_cluster.cc:1428] Setting key aae0a2101bbaab3176ab5bfeab1b146a
I20260928 12:35:16.481411 31103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:35:16.483825 31103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:35:16.491334 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.484561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.8.13.62:44853)
0928 12:35:16.484619 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.484621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.484765 (+ 144us) server_negotiation.cc:407] Connection header received
0928 12:35:16.484846 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.484847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.484858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.484920 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.485327 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.485904 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.486688 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.486960 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.487034 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.488969 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.488971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.488971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.488979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.490194 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.490534 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.490535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.490536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.490588 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.490787 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.490788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.490788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.490905 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.490961 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.491155 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.491238 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:35:16.491604 31107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.484496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.62:44853 (local address 127.0.0.1:60598)
0928 12:35:16.484664 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.484704 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:16.484778 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:16.484939 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:16.484941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:16.485188 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:16.485257 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.485259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.485974 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.485976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.486624 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.486626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.487055 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.487056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.487105 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.487526 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:16.487533 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:16.488890 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:16.490271 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.490278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.490281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.490469 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.490641 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.490641 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.490642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.490730 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.490959 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:16.490966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:16.491050 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 12:35:16.491130 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 12:35:16.491311 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260928 12:35:16.491603 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.481717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.8.13.60:34037)
0928 12:35:16.481965 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.481975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.482007 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:35:16.482625 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.482635 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.482708 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.482863 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.483464 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.484702 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.485591 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.486099 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.486205 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.488578 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.488589 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.488589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.488597 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.490231 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.490556 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.490559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.490560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.490644 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.490818 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.490819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.490820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.491038 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.491172 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.491254 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.491383 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":199,"thread_start_us":62,"threads_started":1}
I20260928 12:35:16.492125 31103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.62@KRBTEST.COM
I20260928 12:35:16.498769 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.492423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.8.13.61:46165)
0928 12:35:16.492463 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.492465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.492544 (+ 79us) server_negotiation.cc:407] Connection header received
0928 12:35:16.492603 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.492604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.492615 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.492661 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.492902 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.493379 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.494124 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.494350 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.494409 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.496339 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.496341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.496341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.496348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.497622 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.497991 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.497994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.497994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.498057 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.498273 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.498273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.498274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.498427 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.498492 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.498626 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.498689 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:35:16.498805 31107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.492360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.61:46165 (local address 127.0.0.1:36426)
0928 12:35:16.492489 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.492499 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:16.492545 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:16.492663 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:16.492664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:16.492796 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:16.492836 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.492838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.493461 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.493462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.494047 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.494049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.494410 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.494411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.494469 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.494874 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:16.494878 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:16.496252 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:16.497701 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.497702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.497703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.497927 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.498122 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.498123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.498123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.498221 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.498488 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:16.498489 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:16.498538 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 12:35:16.498606 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:35:16.498707 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 12:35:16.499130 31103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:35:16.499922 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.492416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.8.13.62:44853)
0928 12:35:16.492456 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.492457 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.492481 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:35:16.492539 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.492539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.492549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.492600 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.493054 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.493513 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.494275 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.494472 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.494525 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.496960 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.496962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.496962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.496971 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.498467 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.499005 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.499006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.499007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.499069 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.499415 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.499416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.499416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.499529 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.499698 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.499733 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.499806 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.60@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.61@KRBTEST.COM
I20260928 12:35:16.505340 31107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.499389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42126)
0928 12:35:16.499448 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.499457 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:16.499498 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:16.499588 (+ 90us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:16.499588 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:16.499712 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:16.499742 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.499743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.500461 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.500462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.501055 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.501058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.501484 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.501485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.501534 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.501963 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:16.501968 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:16.503129 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:16.504459 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.504460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.504461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.504643 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.504821 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.504822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.504822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.504887 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.505091 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:16.505092 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:16.505137 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 12:35:16.505205 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:35:16.505273 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 12:35:16.505437 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.499419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.8.13.60:34037)
0928 12:35:16.499452 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.499453 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.499483 (+ 30us) server_negotiation.cc:407] Connection header received
0928 12:35:16.499535 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.499536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.499545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.499588 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.499800 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.500404 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.501120 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.501404 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.501465 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.503189 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.503190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.503191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.503197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.504389 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.504714 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.504715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.504715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.504765 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.504946 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.504947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.504947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.505036 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.505079 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.505256 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.505327 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 12:35:16.507061 31103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f: Bootstrap starting.
I20260928 12:35:16.507726 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.500639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.8.13.61:46165)
0928 12:35:16.500688 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.500690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.500697 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:35:16.500716 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.500717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.500726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.500783 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.501177 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.501789 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.502444 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.502636 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.502688 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.505138 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.505139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.505140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.505145 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.506604 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.506967 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.506968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.506969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.507026 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.507221 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.507222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.507223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.507353 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.507514 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.507554 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.507634 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:35:16.508064 31103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:16.508355 31103 log.cc:824] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:16.509073 31103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f: No bootstrap required, opened a new log
I20260928 12:35:16.509073 8244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:35:16.510635 31103 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } }
I20260928 12:35:16.510126 30927 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60604
I20260928 12:35:16.510761 31103 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf19cf3082fb4f6d8a3b4138e08b8e7f, State: Initialized, Role: FOLLOWER
I20260928 12:35:16.510838 31103 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } }
I20260928 12:35:16.511124 31103 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:16.511091 31109 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } } }
I20260928 12:35:16.511181 31109 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:16.512140 30953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } attempt: 2
W20260928 12:35:16.512545 31120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:35:16.512598 31120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:35:16.512972 31035 master_runner.cc:186] Error getting master registration for 127.8.13.62:44853: OK, instance_id { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" instance_seqno: 1790598916168626 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:35:16.515805 31035 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.62@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
I20260928 12:35:16.519519 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.512421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42142)
0928 12:35:16.512463 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.512476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:16.512525 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:16.512701 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:16.512702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:16.512872 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:16.512919 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.512922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.513766 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.513767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.514583 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.514586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.515058 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.515059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.515115 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.515586 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:16.515591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:16.516830 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:16.518510 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.518511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.518511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.518712 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.518892 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.518892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.518893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.518944 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.519219 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:16.519220 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:16.519272 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 12:35:16.519330 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:35:16.519419 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 12:35:16.519523 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.512574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.8.13.60:34037)
0928 12:35:16.512610 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.512611 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.512617 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:35:16.512635 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.512636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.512644 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.512696 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.512988 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.513691 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.514681 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.514969 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.515046 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.516910 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.516911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.516911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.516919 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.518432 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.518775 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.518776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.518776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.518833 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.518996 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.518997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.518997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.519153 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.519218 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.519365 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.519438 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:35:16.520992 30953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3: Bootstrap starting.
I20260928 12:35:16.521588 30953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:16.521896 30953 log.cc:824] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:16.522615 30953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3: No bootstrap required, opened a new log
I20260928 12:35:16.523595 30953 raft_consensus.cc:348] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } }
I20260928 12:35:16.523707 30953 raft_consensus.cc:729] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded87bc0301c4cd882e83bf7fdcff1c3, State: Initialized, Role: FOLLOWER
I20260928 12:35:16.523801 30953 consensus_queue.cc:260] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } }
I20260928 12:35:16.524083 30953 sys_catalog.cc:564] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:16.524118 30927 raft_consensus.cc:482] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 12:35:16.524188 30927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:16.524958 30927 raft_consensus.cc:504] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } }
I20260928 12:35:16.525200 30927 leader_election.cc:288] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [CANDIDATE]: Term 1 election: Requested vote from peers a8eb56e2aca641b4b7431f47fd417ef2 (127.8.13.61:46165), cf19cf3082fb4f6d8a3b4138e08b8e7f (127.8.13.60:34037)
I20260928 12:35:16.525650 31077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f"
W20260928 12:35:16.526037 31134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:35:16.526082 31134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:35:16.525779 31077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:16.527450 31077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded87bc0301c4cd882e83bf7fdcff1c3 in term 1.
I20260928 12:35:16.527704 30904 leader_election.cc:302] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf19cf3082fb4f6d8a3b4138e08b8e7f, ded87bc0301c4cd882e83bf7fdcff1c3; no voters:
I20260928 12:35:16.526355 30885 master_runner.cc:186] Error getting master registration for 127.8.13.61:46165: OK, instance_id { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" instance_seqno: 1790598916324658 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:35:16.528945 30885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:35:16.530263 31123 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } } }
I20260928 12:35:16.530350 31123 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:16.530404 31123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:35:16.530454 31123 raft_consensus.cc:686] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [term 1 LEADER]: Becoming Leader. State: Replica: ded87bc0301c4cd882e83bf7fdcff1c3, State: Running, Role: LEADER
I20260928 12:35:16.530521 31123 consensus_queue.cc:237] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } }
I20260928 12:35:16.530985 31123 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ded87bc0301c4cd882e83bf7fdcff1c3. Latest consensus state: current_term: 1 leader_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } } }
I20260928 12:35:16.531072 31123 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:16.537993 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.530028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.8.13.62:44853)
0928 12:35:16.530070 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.530073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.530777 (+ 704us) server_negotiation.cc:407] Connection header received
0928 12:35:16.530920 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.530921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.530943 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.530973 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.531863 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.532322 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.533312 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.533624 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.533681 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.535170 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.535172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.535172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.535196 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.536767 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.537162 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.537163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.537164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.537219 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.537396 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.537397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.537397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.537552 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.537651 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.537821 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.537897 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:35:16.546130 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.530251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.8.13.60:34037)
0928 12:35:16.530305 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.530307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.538030 (+ 7723us) server_negotiation.cc:407] Connection header received
0928 12:35:16.538379 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.538381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.538394 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.538422 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.539034 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.539774 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.540762 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.541063 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.541105 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.542861 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.542863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.542864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.542872 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.544839 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.545229 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.545230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.545231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.545287 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.545479 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.545480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.545480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.545619 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.545695 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.545941 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.546029 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:35:16.546381 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.530030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.8.13.61:46165)
0928 12:35:16.530071 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.530073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.538073 (+ 8000us) server_negotiation.cc:407] Connection header received
0928 12:35:16.538269 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.538271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.538284 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.538323 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.539302 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.539972 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.541281 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.541548 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.541592 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.543482 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.543495 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.543496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.543504 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.544928 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.545408 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.545409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.545410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.545461 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.545705 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.545706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.545706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.545858 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.545922 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.546205 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.546300 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:35:16.546557 31145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:35:16.546785 31145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:35:16.548689 31077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [term 1 FOLLOWER]: Refusing update from remote peer ded87bc0301c4cd882e83bf7fdcff1c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:35:16.549036 31140 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:35:16.550437 31109 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ded87bc0301c4cd882e83bf7fdcff1c3. Latest consensus state: current_term: 1 leader_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } } }
I20260928 12:35:16.550513 31109 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:16.558090 31028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } attempt: 3
Sep 28 12:35:16 dist-test-slave-g31s krb5kdc[30872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.61@KRBTEST.COM for kudu/127.8.13.60@KRBTEST.COM
I20260928 12:35:16.563063 31140 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } } }
I20260928 12:35:16.563184 31140 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:16.563522 31146 mvcc.cc:205] Tried to move back new op lower bound from 7334293162180251648 to 7334293162109677568. Current Snapshot: MvccSnapshot[applied={T|T < 7334293162180251648}]
I20260928 12:35:16.564057 31145 catalog_manager.cc:1347] Generated new cluster ID: 45a00165f6cc41b7bb0c2ea23fe0d8b5
I20260928 12:35:16.564199 31145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:35:16.564313 31109 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } } }
I20260928 12:35:16.564399 31109 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:16.564471 31147 mvcc.cc:205] Tried to move back new op lower bound from 7334293162180251648 to 7334293162109677568. Current Snapshot: MvccSnapshot[applied={T|T < 7334293162180251648}]
I20260928 12:35:16.568531 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.558438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.60:34037 (local address 127.0.0.1:42166)
0928 12:35:16.558493 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:35:16.558505 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:35:16.558546 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:35:16.558684 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:35:16.558684 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:35:16.558871 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:35:16.558924 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.558926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.559897 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.559898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.560700 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:35:16.560704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.561172 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.561174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:35:16.561228 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.561774 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:35:16.561780 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:35:16.563310 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:35:16.567423 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.567425 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.567425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.567649 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.567868 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:35:16.567868 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:35:16.567869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:35:16.567918 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:35:16.568213 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:35:16.568214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:35:16.568264 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 12:35:16.568319 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 12:35:16.568426 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:35:16.568956 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:16.558478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.8.13.60:34037)
0928 12:35:16.558517 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.558519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.558529 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:35:16.558614 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.558615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.558625 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.558675 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.559028 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.559796 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.560787 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:16.561093 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:16.561162 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:16.563404 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:35:16.563406 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:35:16.563406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:35:16.563414 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:35:16.567307 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.567737 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.567738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.567738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.567814 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:35:16.567988 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:35:16.567989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:35:16.567989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:35:16.568131 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:35:16.568186 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:16.568790 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 12:35:16.568869 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:35:16.570935 31028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2: Bootstrap starting.
I20260928 12:35:16.571697 31028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2: Neither blocks nor log segments found. Creating new log.
I20260928 12:35:16.571998 31028 log.cc:824] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:35:16.578714 31028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2: No bootstrap required, opened a new log
I20260928 12:35:16.580719 31028 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } }
I20260928 12:35:16.580889 31028 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8eb56e2aca641b4b7431f47fd417ef2, State: Initialized, Role: FOLLOWER
I20260928 12:35:16.580958 31028 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } }
I20260928 12:35:16.581297 31028 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:35:16.581974 31001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:35:16.582858 31001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [term 1 FOLLOWER]: Refusing update from remote peer ded87bc0301c4cd882e83bf7fdcff1c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:35:16.583081 31002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a8eb56e2aca641b4b7431f47fd417ef2"
I20260928 12:35:16.583254 31123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:35:16.583788 31002 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded87bc0301c4cd882e83bf7fdcff1c3 in term 1.
I20260928 12:35:16.585894 31152 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } } }
I20260928 12:35:16.586004 31152 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:16.586059 31152 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ded87bc0301c4cd882e83bf7fdcff1c3. Latest consensus state: current_term: 1 leader_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } } }
I20260928 12:35:16.586102 31152 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:16.586127 31152 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } } }
I20260928 12:35:16.586176 31152 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:35:16.590363 31168 catalog_manager.cc:1259] Loaded cluster ID: 45a00165f6cc41b7bb0c2ea23fe0d8b5
I20260928 12:35:16.590413 31168 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2: loading cluster ID for follower catalog manager: success
W20260928 12:35:16.590494 31168 catalog_manager.cc:1576] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260928 12:35:16.590526 31168 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:35:16.608073 31123 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded87bc0301c4cd882e83bf7fdcff1c3" member_type: VOTER last_known_addr { host: "127.8.13.62" port: 44853 } } peers { permanent_uuid: "a8eb56e2aca641b4b7431f47fd417ef2" member_type: VOTER last_known_addr { host: "127.8.13.61" port: 46165 } } peers { permanent_uuid: "cf19cf3082fb4f6d8a3b4138e08b8e7f" member_type: VOTER last_known_addr { host: "127.8.13.60" port: 34037 } } }
I20260928 12:35:16.608214 31123 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:35:16.609730 31145 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:35:16.610502 31145 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:35:16.624760 31145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ded87bc0301c4cd882e83bf7fdcff1c3: Generated new TSK 0
I20260928 12:35:16.625057 31145 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:35:16.657284 31032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.655762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.8.13.61:46165)
0928 12:35:16.655808 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.655810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.656606 (+ 796us) server_negotiation.cc:407] Connection header received
0928 12:35:16.656707 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.656708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.656719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.656748 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.657208 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36460: BlockingRecv error: recv got EOF from 127.0.0.1:36460 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 12:35:16.658236 30956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.655981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.8.13.62:44853)
0928 12:35:16.656020 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.656021 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.657714 (+ 1693us) server_negotiation.cc:407] Connection header received
0928 12:35:16.657834 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.657836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.657846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.657875 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.658161 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60618: BlockingRecv error: recv got EOF from 127.0.0.1:60618 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 12:35:16.658972 31106 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.656332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.8.13.60:34037)
0928 12:35:16.656369 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.656371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.658511 (+ 2140us) server_negotiation.cc:407] Connection header received
0928 12:35:16.658601 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.658602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.658611 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.658637 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.658914 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42182: BlockingRecv error: recv got EOF from 127.0.0.1:42182 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 12:35:16.661360 31032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.660566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.8.13.61:46165)
0928 12:35:16.660598 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.660600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.660816 (+ 216us) server_negotiation.cc:407] Connection header received
0928 12:35:16.660914 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.660915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.660932 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.660958 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.661316 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36476: BlockingRecv error: recv got EOF from 127.0.0.1:36476 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 12:35:16.664238 31032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.663508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.8.13.61:46165)
0928 12:35:16.663539 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.663541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.663786 (+ 245us) server_negotiation.cc:407] Connection header received
0928 12:35:16.663872 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.663873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.663883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.663908 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.664197 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36492: BlockingRecv error: recv got EOF from 127.0.0.1:36492 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 12:35:16.668283 31032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.667515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.8.13.61:46165)
0928 12:35:16.667546 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.667548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.667809 (+ 261us) server_negotiation.cc:407] Connection header received
0928 12:35:16.667898 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.667899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.667908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.667934 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.668239 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36502: BlockingRecv error: recv got EOF from 127.0.0.1:36502 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:35:16.674283 31032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.673412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.8.13.61:46165)
0928 12:35:16.673447 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.673449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.673727 (+ 278us) server_negotiation.cc:407] Connection header received
0928 12:35:16.673856 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.673857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.673868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.673896 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.674234 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36512: BlockingRecv error: recv got EOF from 127.0.0.1:36512 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:35:16.684433 31032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.683597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.8.13.61:46165)
0928 12:35:16.683639 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.683641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.683869 (+ 228us) server_negotiation.cc:407] Connection header received
0928 12:35:16.683961 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.683963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.683974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.684002 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.684376 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36516: BlockingRecv error: recv got EOF from 127.0.0.1:36516 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:35:16.702384 31032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.701698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.8.13.61:46165)
0928 12:35:16.701741 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.701743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.701907 (+ 164us) server_negotiation.cc:407] Connection header received
0928 12:35:16.701958 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.701959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.701970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.702015 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.702324 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36528: BlockingRecv error: recv got EOF from 127.0.0.1:36528 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 12:35:16.736109 31032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.735421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.8.13.61:46165)
0928 12:35:16.735475 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.735477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.735556 (+ 79us) server_negotiation.cc:407] Connection header received
0928 12:35:16.735618 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.735619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.735631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.735712 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.736047 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36532: BlockingRecv error: recv got EOF from 127.0.0.1:36532 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 12:35:16.801946 31032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.801312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.8.13.61:46165)
0928 12:35:16.801383 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.801386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.801444 (+ 58us) server_negotiation.cc:407] Connection header received
0928 12:35:16.801492 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.801493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.801503 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.801564 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.801885 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36548: BlockingRecv error: recv got EOF from 127.0.0.1:36548 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 12:35:16.931598 31032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:16.931036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.8.13.61:46165)
0928 12:35:16.931098 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:16.931100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:16.931111 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:35:16.931162 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:16.931162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:16.931172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:16.931227 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:16.931534 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36554: BlockingRecv error: recv got EOF from 127.0.0.1:36554 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 12:35:17.189344 31032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:35:17.188700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.8.13.61:46165)
0928 12:35:17.188763 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:17.188764 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:17.188778 (+ 14us) server_negotiation.cc:407] Connection header received
0928 12:35:17.188847 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:17.188848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:17.188858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:17.188908 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:35:17.189285 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36562: BlockingRecv error: recv got EOF from 127.0.0.1:36562 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:35:17.513046 31120 catalog_manager.cc:1259] Loaded cluster ID: 45a00165f6cc41b7bb0c2ea23fe0d8b5
I20260928 12:35:17.513123 31120 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f: loading cluster ID for follower catalog manager: success
I20260928 12:35:17.513942 31120 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f: acquiring CA information for follower catalog manager: success
I20260928 12:35:17.514221 31120 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf19cf3082fb4f6d8a3b4138e08b8e7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:35:17.591578 31168 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2: acquiring CA information for follower catalog manager: success
I20260928 12:35:17.591933 31168 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8eb56e2aca641b4b7431f47fd417ef2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:35:17.705013 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:17.702396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.8.13.61:46165)
0928 12:35:17.702526 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:17.702528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:17.702577 (+ 49us) server_negotiation.cc:407] Connection header received
0928 12:35:17.702644 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:17.702645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:17.702698 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:17.702772 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:17.702928 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:17.703456 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:17.704222 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:17.704452 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:17.704517 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:17.704681 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:17.704788 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:17.704853 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:17.704900 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 12:35:17.704922 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":44,"threads_started":1}
I20260928 12:35:17.708726 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:17.706637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.8.13.60:34037)
0928 12:35:17.706735 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:17.706736 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:17.706770 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:35:17.706841 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:17.706842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:17.706888 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:17.706956 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:17.707028 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:17.707509 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:17.708150 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:17.708351 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:17.708398 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:17.708489 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:17.708575 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:17.708621 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:17.708651 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 12:35:17.708665 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":45,"threads_started":1}
I20260928 12:35:17.712944 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:17.710264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.8.13.61:46165)
0928 12:35:17.710387 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:17.710389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:17.710396 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:35:17.710419 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:17.710420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:17.710429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:17.710479 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:17.710603 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:17.711350 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:17.712333 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:17.712559 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:17.712605 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:17.712727 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:17.712756 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:17.712801 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:17.712854 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 12:35:17.712873 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 12:35:17.713320 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:17.710180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.8.13.62:44853)
0928 12:35:17.710276 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:17.710278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:17.710393 (+ 115us) server_negotiation.cc:407] Connection header received
0928 12:35:17.710568 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:17.710569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:17.710616 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:17.710667 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:17.710851 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:17.711600 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:17.712402 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:17.712699 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:17.712756 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:17.712962 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:17.713070 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:17.713109 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:17.713213 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 12:35:17.713236 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":46,"threads_started":1}
I20260928 12:35:17.713538 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:35:17.710375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.8.13.60:34037)
0928 12:35:17.710447 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:35:17.710449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:35:17.710487 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:35:17.710556 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:35:17.710557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:35:17.710566 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:35:17.710610 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:35:17.710733 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:17.711227 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:17.712817 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:35:17.713091 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:35:17.713140 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:35:17.713298 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:35:17.713328 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:35:17.713375 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:35:17.713452 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 12:35:17.713473 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:35:17.714999 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 30885
I20260928 12:35:17.719825 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 30960
I20260928 12:35:17.724148 8244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2Ofqh2/build/release/bin/kudu with pid 31035
2026-09-28T12:35:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2728 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5482 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277709 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST